Yes, but it's not illegal for a person to have an affair. It's not illegal to watch pornography. It's not illegal for single people to have sex.
So this leads back to the same discussion, which is whether it's fair to legislate gay marriage. Not whether it's fair to agree with it, but whether it's fair to insert yourself into the decision.