On Bypass Rd., about 5 min away from Colonial WM is a Cracker Barrel. :) For us, it turned into a lifesaver because we had some really bad dining experiences that week.
In Colonial Williamsburg, we enjoyed eating at Shield's Tavern. You can ask to be seated on the lower level which is lit only by candles so it was eerie and cool at the same time.