Rather than avoid religion altogether, perhaps this is a time for your son to learn about many different beliefs, including yours. My husband and I are devout in our beliefs, but we strategically allow our 4yo to learn about what others believe and to respect those other beliefs. For example, we always pray at mealtime, and when she notices others not doing so, we discuss with her why other people might not pray in a way that allows her (we hope) to respect others' choices and beliefs.
So if you want your child to have a religious education, why not study many religions and take some time to compare and contrast them? Help him understand why you don't believe some things that he has been taught. I don't think you have to have all the answers to provide a religious education to your children.