As far as Xander and Anya are concerned, my theory around them is that, as far as Xander is concerned, Anya never did anything personally to him or any of his friends. It's all secondary and collateral damage, and he doesn't even know what happened in 'The Wish' anyway, so he's able to look past it all more.
Spike, Faith and Andrew were all, at one point, laser focused on hurting Buffy and her friends, which included Xander, so I think, for him, it's harder for him to see past that. Although, he seems to have gotten over it with Andrew, maybe because of Warren's 'punishment' since he was the ring leader.
As far as I'm concerned, out of everyone, Andrew is the only one who actually hasn't had some sort of redemption arc. His 'moment' in 'Storyteller' isn't really what I'd call redemption, he's just scared. Plus, he hasn't really actually moved on or grown up any, considering what he does to Buffy in S9. We're always all over Spike for the AR - and quite rightly - but Andrew literally stole Buffy's body without her consent and nothing has ever been said or done about it. To me, he needs to pay some sort of penance for that, or at least become aware that it was the wrong thing to do. As of right now, I don't believe that's happened.
Everyone but Andrew has shown some sort of remorse or guilt over what they'd done. They all pay for their crime in some way, even if they do get over it pretty quickly. Andrew never has, he just jumped onto Buffy's ship because he had nowhere else to go and the end of the world was pretty much nigh. He had nothing to protect him and he tied himself to Buffy's mast out of nothing more than desperation. I still find it hard to believe that Buffy trusted him enough to assign him a squad, never mind what actually happened to said squad.
Andrew might seem innocent, but, to me, he's actually one of the most dangerous people in the verse. The other Big Bads are monsters and we know it, Andrew is secretive and capable of horrible things. Maybe Buffy is keeping him on her side by keeping him sweet because she doesn't like to think about what he could do if he goes dark side again?