I feel you pain. Mine are 7, 4, and 2. My aunt, who homeschooled her 5 kids over the course of about 20 years, said that having a toddler in the house is absolutely the most difficult aspect of homeschooling. At my house, we grab bits and pieces of time throughout the day and just do what we can. When I notice the 2 year old being relatively happy, I pull out something that takes 20 minutes or less and do that. I rotate what I start with so the same subjects don't get neglected over and over. I continue working if we can, but if not I just try to grab additional 10-20 minute snippets as the day goes on. The 2 year old will sometimes listen in on our read alouds. One thing that has helped me feel better about things is Explode the Code online. I know that my 1st grader is working at least 20 minutes on phonics every day no matter how crazy things are. We sometimes work during naptime or I will turn on the TV for the 2 year old too.
ETA: A good attitude during schoolwork is required at my house for games to be allowed during that day. Repeatedly asking for games during schoolwork doesn't count as a good attitude even if the asking is done nicely.