I'd like to broaden this a bit to say success in acting isn't just from the audience perspective - it's also whether it fulfills the actor's desires. ASH has done a few shows with the BBC, and is well like there at home even if we don't see uof him here. Amber Benson writes children's book and direct/produces small independent films because she didn't like the Hollywood experience. Marsters does mainly music and stage acting - last I read he was playing Macbeth in some LA theater. (again the Hollywood thing - in fact he's done a few films with Amber). Denisoff is on Finding Carter - so he and AH might have decided to trade off on which parent stays with the girls.
I have a feeling that a long run on a tv show can be exhausting - witness SMG singlehandedly cancelling Buffy in her VF interview. Taken from the actor's needs, they may well be happier without that kind of success. (I particularly like AB!)
To me, for better or for worse, I think it can also come down to a fair degree of luck or things that are just well outside of the control of the actors and actresses involved.
Taking Mr. Boreanaz into consideration, while I haven't watched as much of Bones as I have of Buffy or Angel, I personally think that overall he's done an excellent job in both of his roles over the years. That being said, one could say a fair bit of luck went into it, some of which was beyond his control.
Sad as it may be for us of the Buffyverse fandom, we never got that 6th season of Angel, something that Mr. Boreanaz and the other fine actors and actresses on Angel could do nothing about, the Powers that Be having already made their decision. OTOH, suppose a sixth season had been approved. This is good for the Buffyverse fandom, unless it turns out to be utterly terrible or something, and at the time it would seem good for Mr. Boreanaz and his other cast members. That being said, a sixth season would have started production in the summer and fall of 2004, and probably wouldn't have finished production until well into 2005. Bones first premiered in September of 2005. Had Angel gotten a sixth season, not taking into consideration one way or another a 7th season, if he was so busy with Angel it's not impossible that Mr. Boreanaz might not have had the time available to audition for his role in Bones, or after finishing a 22 episode season 6 of Angel he might not have wanted to do anything else immediately even if he was available for auditions at the time.
End result, regardless of whether or not a Bones without Mr. Boreanaz playing Booth is successful or not, there's not telling what might or might not have happened to his career post Angel had he not landed the role on Bones. For all we know he could have an Oscar or two right now, or post Angel he could have faded into obscurity, making an appearance on the convention circuit occasionally possibly but otherwise dropping out of the limelight.
OTOH, a sixth season, 22 more episodes. Depending on who else among the cast is willing to sign on or not for a sixth season, this may or may prove a major shot in the arm for some people. Andy Hallett probably wouldn't come back as Lorne due to his medical problems that cropped up shortly after the end of season 5 that tragically led to his untimely death in 2009, so either other main cast characters get more screen time between them, or another high profile recurring character is promoted to main cast perhaps. For a perhaps lower profile actress like Mercedes McNab, having a full 22 episode sixth season as a main cast member could be great for her career. Not that she was bad earlier, but getting a chance to fully step into the spotlight for an entire season, give her character some more depth, explore her life, etc, depending on how she handles it, who knows, maybe it doesn't change her post Angel career all that much. Maybe she does pick up one or two more guest or recurring roles here and there, you never know. Given that after 2004 she didn't do anything else until 2006, picking up another 22 episodes worth of a good tv show and carrying her into 2005 as a main cast member could easily help boost her into landing a couple of additional guest spots her and there at least.
And I won't even get into the possibility of other Buffy actors and actresses that could have popped up again, even if it was only a couple of people for a one show guest bit or something.
