I'm raising a trilingual, it was a challenge at first, now he is 9 and fluent, so don't give up! You can do it. Speak to your kids in French only, just like you said, but don't limit yourself to few hours, jump head first and do it all day long. We did it and we are happy now.
We also expect my son to reply in the language we spoke to him, if he doesn't, we don't understand him, period. He is allow to draw, describe, or get a diccionary, but we don't bend that rule because It works great around his "wants" and you can see results fast. You may have to endure a week of meltdowns, but if you are consistent, they will respond and you will see amazing progress.
Don't forget to play music, funny lyrics are a plus ;)
Good luck!