I can't imagine.
I'm not very good at being direct and I always worry about hurting people's feelings. Sounds like you might be the same. I bet once you say "no" one time you will feel more empowered and maybe they will get the hint.
Could you try saying something like, "I'm sorry but I want my son to stay home today so we can have some family time." Once you say it, just keep saying it more frequently. Have them over but limit his going over there unless you are invited too.
Good luck.