Gavin really bugged me, but then he wasn't a main character in my mind so i'll let that go.
like Dzhug above, i felt that Lorne got a bit pointless as the show went on...it was like, by the time mid-season 4 hit, they ran out of ideas for him.
all in all, there are moments for each character where they got on my nerves when they did certain things, but none of the main characters ever annoyed me to the point where i was yelling 'just kill him off already!' i liked everyone in their own way.
and on one side note, not that either of them could EVER annoy me in the least, but why aren't Spike and Illyria part of this list? and for that point, Harmony?