My kids are older, like yours, and it wouldn't work for me. I'm still too involved in each of their schooling. My day is spent going from one student to the next to help or go over various things. I might have a few breaks here and there but my focus during the day is them and their school work. Only you know how much time you spend each day interacting with your students. How would that interaction change if there was an infant to care for?
Obviously, if it were my own baby I'd make it work and be happy about it :D . But when it's for money, I would want to think through how it would affect me and my family.