Well, as has been mentioned, there's the whole Tara thing, which may have scarred me more than anything else I've seen on TV. Granted, it was probably one of the ONLY unintentional things that he did....but still.
I took most of my issues with how he treated Andrew and Jonathan, (and perhaps Katrina and the robot girl.) How he treated Buffy, Willow and society at large wasn't good, but it was that he had NO loyalty to even the people who were supposed the be the ones close to him....it really leaves nothing to like.