I have an only. It was difficult the first few years. And we used to do a lot of activities, trying to find our way. It paid off, because in time, through those activities, we met our own small circle of friends. My son is very social, and was at one point so lonely that I worried about him a lot. He seems content, now.
I'm from a big family, and I love big families. I love our small families. And I've seen some snarky, offensive comments about both extremes. Seems to me, whatever choices you make, there are people that will approve and people who won't. You aren't going to please everyone. So you do what you think is best and shrug it off.
Also, I am a person who regularly says things that I regret later, so I have sympathy for others who may not understand the effect of what they're saying.