I don't think Boreanaz is that good of an actor, but most of the actors on the show weren't that good. Alyson Hannigan is a horrible actor- she's good for her niche, and that's about it. The same can be said for Boreanaz. It just depends on what type of one-note actor you favor, because honestly, with few exceptions that is all that are in the show. Some of them aren't even acting merely playing themselves- Carpenter, McNab spring to mind.
Not sure I'd go as far as saying Carpenter and McNab are
just playing themselves, but then I don't know much about the actresses to know how closely their characters resemble themselves. Certainly their roles aren't the most demanding or varied; Harmony is of course one-note but she's supposed to be because that's what makes her funny. I think CC does quite well with Cordy's initial growth from bitchy-shallow to bitchy/snarky-compassionate but she struggled with more (like
@dcai0830 said, 'in-love' Cordy was unconvincing, and Evil!Cordy was just dire). I think you're right about most of the actors not being 'that good', in the sense that they're not all-rounder 'Actors' with a capital A who can handle a lot of variation, but what's important is that they do fit the roles they're cast in, so being good all-round actors doesn't matter too much. I haven't seen HIMYM so I can't judge Alyson on that, but on
Buffy she
is Willow and I adore her, so there's no way I'd call her a terrible actress.
Of the female cast I think Sarah was wonderful and showed incredible range. I'd have to watch her performance more carefully but I think one weak thing might have been the mock-tears she sometimes had to do for the lighter episodes, which sometimes seems forced (I'm thinking of her 'shoulder-shrug' cry in 'Life Serial', which is sitcom-funny, but felt like SMG had to try hard for it). But again I adore her and she's brilliant 99% of the time so that's really minor. Eliza was just fantastic as Faith and I think Amy showed the most skill, especially in that transition from Fred to Illyria, which was just - wow!
Of the guys Tony, James and Alexis were the most technically accomplished and that's to be expected as they all come from theatre backgrounds and are classically trained. That comes through naturally in their performances which are always solid yet nuanced. Then of course as Spike's role was expanded it allowed James to bring in more variation and depth.
he didn't have to carry the romantic melodrama and could be the brooding vamp he was intended to be instead of the romantic leading man
The character was never meant to be the leading man. He was a supporting character in Buffy's story and I'd say his role overall was actually tertiary rather than secondary - the main supporting characters were Willow, Xander and Giles. Angel comes after that in the ranks, and only comes to the fore when it's necessary to Buffy's story.
There are scenes of obvious melodrama like the early S2 stuff (Some Assembly Required), but it's Buffy driving it and it fits at that point because she
is a teenager after all so she's allowed to behave that way at least a little. But then there are more quiet down-to-earth moments like 'Earshot' which provide a nice counterpoint to the high drama. I actually like the moment in 'Graduation Day' when he does raise his voice a little during their argument. It brings in a bit more emotional variation and frankly it would be even more unrealistic to expect him to remain steady calm all the time. Just because he is by default reserved doesn't mean he can never get worked up.
I think it would be difficult to do their story without an degree of melodrama because it is written to be 'big, world-in-peril stuff'. The situations are extreme (gypsy curse, world getting sucked into hell - the circumstances are very life-or-death) and the characters have to respond to that in a heightened way for the story to hit a dramatic high note, even if they do mock/snark about it afterwards. That balance is part of the joy of the show. Melodrama actually drives the whole series- without it it just wouldn't work because it uses melodrama via larger-than-life fantasy and horror tropes to access a deeper psychological and emotional realism for its characters.
I think romantic melodrama instinctively elicits an eyeroll response because it is so tiresomely widespread in popular culture and fiction, but on BtVS it more often than not worked because of the nature of the show, which in addition to dealing with larger-than-life plots, was also very soapy.