This was the case with my childhood, as well, though we did have a few alcoholics in the bunch. The rest of the family (my dad's side, my parents, my maternal aunts and uncles, save one of the spouses) would have a drink here and there on occasions, no big deal.
We went out to eat crabs when I was pregnant with the now 11yo, and I had a non-alcoholic beer with them, as did my then teen eldest daughter. It's just part of the ritual. I suspect, in the OP's context, it's also part of the ritual of the men sitting aside together at the gathering.
I understand the concern, but I don't share it.