I think you should think about what it will feel like to YOU to have him there. If it would detract in any way from your day, I wouldn't do it.
The pp poster was right about this opening the door to a relationship with him, at least I think he could take it that way. If you don't want that, I wouldn't invite him.