I feel that way about Barbie dolls, princess toys, and all that other prissy stuff. I had Barbies and they didn't ruin me, but I see nothing positive about their existence. Legos, on the other hand, I LOVE.
I swore my daughter would never own a Barbie. She's four and I'm still winning that battle, but she absolutely loves the princess things and anything she can find that is either pink or sparkly. She has lots of that stuff I didn't want her to ever have.
I've realized that my child is going to pursue the things she's interested in whether I like it or not. There are going to be so many of these things - everything from a lollipop before dinner to dating before age sixteen. I've decided that I'm not going to be the NO mom all the time and so I'm going to have to compromise with my little girl as she explores what interests her. I will put my foot down on the dating before sixteen, but I will have to give in once in a while on the lollipop before dinner. Like Legos, that lollipop won't ruin my child or leave her with a long-lasting negative self-image or negative image of her mother.
Legos won't ruin your child. I know they are annoying to clean up and tragic to step upon, but the flip side is that they also spark imagination (especially if you skip the step-by-step building plans) and teach processes (if you use the plans). They are unisex and multi-functional (I have a stand for an iPod Touch that my son made for me from Legos...it's really the perfect solution for a problem I had, and he figured it out all by himself.) We have used Legos for science, robotics, art, and even language arts. They have positive features, for sure, so if you must compromise somewhere, they're a safe bet. You don't have to completely give in, but maybe sharing with your child your feelings about Legos and teaching him how to compromise (maybe he can have some, but they come with boundaries - maybe they have to stay in his room or something) would actually be a great benefit.
That's just my two-cents. I don't want to be shredded for it - it's just my perspective on an issue all parents deal with!