Exactly, he simply wanted to be with Anya, no fuzz. I think he wanted to spend the rest of his life with her, but it was the marriage bit that freaked him out.
Maybe it shouldn't, but marriage sounds to eternal, so I can see why it freaks people out even though they don't want to actually break up.
If I recall correctly, Xander wanted to get back together with Anya afterwards. So he wanted them to live as marrieds, but he didn't want to be married.