You are right that people seem to have trouble getting along — mostly with anyone that falls into “other” because the brain tends to sort into us and them, which is a prehistoric way of staying safe that’s hardwired into our brains.
I don’t like score-keeping, either. People who say “men” are the problem are perhaps a little lazy with word choice. I suppose it’s shorthand, to a degree, instead of saying “societal definition of what it means to be a man,” which is what the real problem is. Hope that makes some kind of sense. :)