I think Oz's analysis of the situation is correct, Xander and Willow have a very long history together, and you never really fall out of love, especially as an adolescent. Because you're love's bitch. Okay, maybe Oz didn't come up with that part. Point is, unique circumstances.
Xander strikes me as the type who would be "satisfied" with what life has given him, and as his key characteristic is loyalty, would remain loyal to Anya. As for Anya, well, she is a monster. But I also don't want to give Credence to Darkest Timeline magic vision quests, as hate leads to suffering, not power.