We tried a similar schedule this year for first grade. It didn't work for us because:
1) DD was tired after lunch and it was hard to get her to focus (Yours are older, though, so maybe not a factor?)
2) It seemed like something else inevitably came up... Out of town guests, unique field trip opportunities, statuatory holidays (DH was home, so no school), illness, etc... Then we were down to TWO days of school that week...
3) On the days we were heading out, we just couldn't seem to buckle down and get work done... Too busy packing lunches, etc.
If you are a very disciplined person, maybe this wouldn't be a problem. For us, I have decided to schedule 4 days of school, with nothing before 3 pm on those days. That way if we get invited somewhere cool, we still have 3 days left! :)
Heather