I don't think it's considered the most hated. For one thing, I think Killow tops it!
The thing about Briley, (or Ruffy!) is that it's the only of Buffy's long term relationships about which we can say "that didn't work," because there isn't really any hope for them, for the future. Riley is happily married. I think a lot of the time when we battle over "who is better" for Buffy, we are taking into account what we hope will happen, based on events both past and present. Briley gets a lot of flack when seasons 4 and 5 are reviewed, and I think a lot of that is due to the fact that we now know that it didn't work out.