Whedon and the rest wanted Buffy to be the "typical college girl" (in their minds) rather than Buffy. Which meant dumbing her down, and I think ignoring previous experiences with colleges (though there is the plausible argument that Parker--and in this case Riley, for me if I accept this--is rebound guy than real romance, and makes me pity Buffy, and pity Riley even more, in addition to the "obligation" I describe below).
I can't find it so I must not be remembering the quote right (or the transcript is censored somehow). Buffy talks about how perfect Riley is, but just doesn't feel right, while Willow says something "run while you still can." The entire thing came off to me, probably unintentional by the writers, that they were trying to convince her to be with him, because once again, she is nothing without a love interest. Something terribly wrong with you if you're not with a guy, if you're not careful, you'll turn into Faith.
Plus, her abandonment issues were again underscored around this time, which imbues Buffy's "need" for a boyfriend seem like desperation and pain more than actual love and desire (even genuinely finding Riley hot, as opposed to genuinely growing close to him, wouldn't be enough to erase that in my mind).
It felt to me that Buffy didn't WANT to be with a guy but felt OBLIGATED to...and Willow also bought into that (subconsciously). That ruins it for me.
Not the character (I like him more than most--and ironically, he seemed more observant and caring of Willow during her heartbreak than Willow's own friends), but the ship.
Also, Riley punching Parker loses some of it's appeal when I realize that Parker felt safe enough to say that around Riley and the others are surprised when Riley punches him, implying that if he did not know Buffy (or been someone else he wasn't familiar with), he'd been fine with it, at least not worth the trouble of making a scene. Just a little detail, but there are plenty of others like it.