Call me biased, but I think James Marster's performance is the most successful of all cast members on both shows. All of the actors have their wonderful moments--often several of them (Anthony Stewart Head, SMG, Emma Caulfield, and Alexis Denisof are all stand-outs for me), but I feel that JM was the most consistent throughout the course of the series. His performances at the end of "Beneath You" and in episodes like "Afterlife" and "Intervention" amaze me every time I watch. The nuances he uses add so much complexity to the character.
I've got to agree with those who say David Boreanaz is a weak actor. He definitely got much stronger as Buffy progressed (Season 2Angelus was fabulous) and over the course of his own show, but really the only performance of his that truly impressed me was his reaction to Wes taking Connor. That was some intense stuff. DB does angry and evil very well.
I haven't seen Firefly or Serenity yet (can't wait to get my hands on them!), so I'll have to wait to judge those actors.
Oh, and Juliet Landau definitely deserves some credit. Drusilla is awesome.