Bless your heart. Hugs!!!!
If you feel that it's not a good match for him and your family right now take the pressure off and give yourself permission to regroup. I used TOG several years ago and I completely understand. There is no way in our season of life right now that I could make it work for our family and my 10th grader (we moved last year, my best friend passed away this summer from a battle with cancer). When life changes happen...taking care of a baby is definitely one of those changes, cut yourself some slack. There is no shame!!!! I wouldn't try to make it work.
You might consider using a textbook this year (Notgrass, Bob Jones, etc). Easier lesson plans fewer books and then if you want to still add in a biography or other books to read you can. Notgrass has suggested books to read with their history books but not a bunch.
Give your son some options for curriculum (that you have determined to be ones you think will work) and see what he thinks he'd rather do.
You do what is best for your family. Don't load yourself with guilt and what ifs. It will work out fine and make it more enjoyable for both of you.