It’s because Millennials and Gen X have grown up with a Xander in high school.
Also compared to other characters Xander is more ‘bland’. He isn’t a supernatural creature and doesn’t have an ability.
The other thing actually about Xander: he hurts the women he's involved with, really needlessly. Hurts Cordelia by having a sort of affair with Willow, hurts Willow just as much (and as a bonus also hurts Oz for that matter), hurts Anya leaving her at the altar. All selfish, all needless. That's a big part of why I see him as kind of a jerk.
I can think of two things he's done wrong in relationships, both of which are completely normal, human things and understandable (which is not to say right.) He kissed Willow a few times after realizing he has feelings for his lifelong best friend, most notably when he thought the two of them were going to die at the hands of Spike. And he realized he wasn't ready for marriage too late when an external force exposed him to a vision of his personal nightmare—becoming his father and hurting the woman he loves. So his sins are a lack of control over newfound feelings for the person he loves most in the world and fear of being abusive. What a monster. 🙄