I feel like non-Bangels also give him a lot of love... many non-Bangels ship Cangel and really like the way he is in that frienship/relationship.
Also, I don't think we get to see a lot of Angel's character until after he's with Buffy, in his own show. I am a hardcore Bangel shipper, but even I will admit that he doesn't have much personality early-on; I attribute this to bad writing, as I think they wrote him originally with the sole purpose to be Buffy's love interest--I think you can read into a lot of small things to comment on his character, but he is pretty basic on the surface of what they give us until his own show.
As a result, a lot of people actually only like him after that point, therefore only liking him when he's not really a part of Bangel, meaning a lot of people who really like him actually aren't Bangel shippers--when Bangel stuff is happening, not much is really happening for Angel, which I also think this is a writing choice. Buffy's show is all about her and they rarely want her love interest to have enough going on that it would overshadow her story--her SOs develop more later in the show, but like I said, Angel was created solely to be her love interest and he was created in the very beginning before the show got really rich with character development, so he is the one who suffers the most from bad/basic writing choices as Buffy's SO.
Full disclaimer, I am a Bangel stan and I prefer him when he is on Buffy and not his show because I prefer him when his sole (haha, soul/sole) purpose is to love Buffy, BUT my above comments are more an objective analysis of why many others like Angel, because I really understand that perspective, despite the fact that Angel as part of Bangel is mostly the Angel that gets me in my feels. I wish we had seen more of Angel in his own element with a deeper personality having a relationship with Buffy, though! I feel like we got a glimpse of this dynamic in End of Days/Chosen and a little towards the end of season 3--he's more sarcastic and sassy and less of an intense brooding loner when he interacts with Buffy.