Hmm, how about:
"Shh, my husband seems to think so!"
Or, you could just go on the offensive and make that joke before they do. Once you say it ("Am I allowed to have this? You sure?"), it kinda takes all the fun out of it for them. The best part is that your issue is looking young, something most people would die for!