If Xander was gay? Well, then he'd be gay. They probably would have brought on a lover for him like they did with Willow. I do agree, though, that the ratings would have suffered a bit due to homophobia. This was pre-Brokeneck, people, and maybe even pre-Queer as Folk (does anyone know the timeline on that?).
As for Spike and Xander... umm. I can see it, but not as a love thing. More as a sex thing. I mean, they were both pretty randy. And it's been hinted that Spike isn't the straightest guy in the world (hints of a Spike and Angelus past, especially). As for Xander, well, he's been known to follow his hormones more than his heart. Kind of like Xander and Willow. Or Xander and Cordy. Or Xander and Faith. Or even Xander and Anya, in the beginning. Xander was sort of horny, yeah, but he didn't fall in love easily. It would have taken a LOT of time to make Xander and Spike fall in love - time that the series didn't have if it wanted to explore Anya's character and Spike's character as ex-demon moving toward-the-human-side type people. I guess what I'm saying is that even though I could see a gay Xander possibly having sex with Spike, I can't see them being a couple.