I use My Father's World which incorporates SOTW into their program. They recommend folding in the other children wherever you are in the history cycle, but they do address the issue with SOTW 4. They felt that material was not suitable for children under 4th grade so they designed a 2nd-3rd grade supplement to go along with SOTW 4 along with activity sheets. (I've also heard complaints that SOTW AG doesn't have as many pages for the younger ones to use so this would solve that problem as well).
Even if you don't want to do their whole program you might get some ideas from looking at their supplement and the sample of their schedule. They have the schedule for what the older ones and younger ones do on the same page.
http://www.mfwbooks.com
SOTW 4 is in their 1850 to Modern program.
SOTW Volume 1-3 I think are more appropriate for the younger ages and I think it would be fine for everyone to do it together.