If you don't mind I first need to correct some errors in understanding of the AtS villains.
Jasmine
Evil Cordy
The Beast
Wolfram & Hart
Holtz
Jasmine and Evil Cordy are one and the same. Evil Cordy WAS Jasmine (how her evil behavior while using Cordy connects with her saintly disguise once's she's born into her own being is beyond me though).
Then you forgetting villains; we also had Darla and Dru in season 2 (though Darla was a little of both good and evil for her AtS duration) and then Angelus returns again in season 4 as a player in Evil Cordy's crap. Then, though I've not seen season 5, I think I've read that The First actually plays a role in it, correct me if I'm wrong.
Both series had their well developed and gripping villains but also their ridiculous nonsense villains.
In BTVS I'd have to say my top favorites are:
The Master - Even though we don't really know that he's somewhere between a primitive and modern vampire until the primitives are directly introduced in season 7, he just has this classic Nosferatu feel to him. The whole thing with the anointed one was idiotic, but I can get passed that.
Mayor Wilkins - A human man who sold his soul to become pure demon was pretty attention grabbing especially with the revelation that Schnider was working for him the whole time, that made a world of sense of Schnider's attitude towards Buffy. I also enjoyed the whole arc where he became Faith's ideal father figure.
The initiative and Adam - There were a lot of great things going on here, basically a modern-day Frankenstein's lab fused with the concept of the short-lived series
Dark Angel (It sadly only lasted two seasons but was a well-done series, I recommend checking it out). The idea of a military operation experimenting to make the perfect soldier by combining human, robot and demon is totally intriguing. The concept of Adam was well developed and kind of symbolic. Once a young man who met a tragic end brought back as the first,
the Adam of the new Demonoid hybrid species they've been experimenting with. That his first act was to kill his creator in a way could even represent Adam and Eve's betrayal in Eden.
My favorites in AtS:
Holtz: To an extent. I hated the whole 'the father will kill the son' thing, but the whole backstory of the Holtz character and the concept of a vengeful vampire hunter was kinda unique to this franchise. While I didn't quite understand the logic behind the whole Austin Powers, suspended animation act the whole support group for victims of vampires was pretty interesting. I'll give it that how as a result he had to face Angel now as opposed to Angelus was an interesting contrast. I remember being on the edge of my seat waiting for him to discover Angel's soul because I was anxious to see his reaction. Then I must admit though I can't stand Teenage Connor, turning Angel's old child against him was kind of a brilliant act on Holtz's part.
Wolfram and Hart: Although it's not original for demons to "hide behind man's law" as the Oracle in 'To Shanshu in LA' once put it and they weren't fully developed because it was never made clear why they wanted to turn Angel back into Angelus or what they're ultimate goal was, these guys were well organized, unpredictable and infuriating. Plus you got this whole conflicted thing with the Lindsey character. First he's an asshole then suddenly he's this tortured artist whom you want to sympathize with...oh wait... never mind he's an asshole again. I've got to admit Christian Kane does have a great singing voice.
Now the one who makes the top of my list in BOTH series,
Angelus! His sick, twisted nature is just so darkly poetic. The whole thing with him and Evil Cordy in season 4 of AtS was stupid but before that, the whole Hannibal thing with him in the cage was brilliant and likewise for his whole duration in BTVS. 'Passion' was my favorite Angelus episode in BTVS for that whole other side of the coin of his obsession with Buffy. The sickness of it, the chillingly Oscar-winning performance as Angel and his monolog at the end, I know I've used this word already but...BRILLIANT!
Think they tried to go in a different direction with the trio but for me it bombed. Those are just supporting characters they should not be taking up so much screen time.
I agree, the trio was ridiculous. You know how in 'Normal Again' the doctor in the institution described them as being a sign that Buffy's delusion world was growing weak? That's exactly what I think it was concept wise, Whedon's creativity progressively weakening beginning with Glory. Though The First in 7 is kinda intriguing so far (I haven't finished it).