Angel, Spike, and Anya are, throughout the series and before, exist in various states of demonic-ness and humanity. It's because of that "grey area" that they all possess that makes us more or less able to stomach the notion of any human being falling in love with either a current demon who does not kill or a former demon who did kill, on any level.
Anya wasn't truly a human throughout her relationship with Xander. Sure, physically she was, but in every other way that counted, she wasn't simply a human--it was far more complicated than that. For 1100 years she was a demon, by choice, (already this is a huge count against her when compared to Angel, Spike and all other vampires), and then suddenly she wasn't, and not by choice. Almost immediately after her powers were taken away from her, she and Xander started dating. So were those Xander's parameters? As long as someone was physically human, regardless of what they'd done almost literally the day before, and in spite of where their values might actually lie, (we know Anya hadn't fully given up her demonic perspectives until season 7), they were OK to date? Yet a vampire who either won't or can't harm anyone, and actively does good things for Buffy and the other scoobies is automatically "off limits" because of his composition? Personally I think it's more complicated than that, but more on that later.
Clearly, the parallels here make Xander hypocritical. To me, it's immediately obvious that Angel is a more suitable choice over Anya when it comes to having the "moral high ground" regarding partner choice; when Buffy is with him he's a hero who chooses to save lives and repent for his past, whereas Anya is merely accepting--begrudgingly, even-- of her new status as human. Sure, she's got one up on Angel because she has human parts but is that really where it counts? Then in season 6 there are direct parallels between Spike and Anya; both are unable to harm and both are helpful to the scoobies, propelled by their deep feelings for one member of the group in particular. Sure in season 6 if it weren't for his feelings for Buffy, Spike might be moping in a corner somewhere or hatching all sorts of plots. And what about Anya? If she weren't accepted by Xander, who's to say she wouldn't be out trying to get her power back, so she could continue doing what she'd done for centuries? Oh, that's right. She does do that. As soon as Xander's not a possibility, she throws her supposed, humanistic, newfound sense of morality out the window and goes off to become a demon again, regardless of the fact that she's "human" when she makes this decision. Around the same time, Spike is in Africa getting his soul so that he can truly be a good man. Again, if we're going to play "moral high ground" here, who's the better partner?
The point is: the simple fact that Anya was technically human while dating Xander doesn't earn him, in my opinion, a pass when it comes to his hypocrisy. That was a merely technical difference. In reality, there were good cases for both avoiding and embracing all three "demons" at various points during Buffy and Xander's relationships with them.
So the hypocrisy is quite clear to me. That being said, because of the aforementioned reality that each relationship had its negative side, it's not at all unreasonable that Xander would be concerned for Buffy's well-being, and for him to express that to her. An alcoholic might be hypocritical in telling his son not to drink so often, but does that mean he shouldn't? He has every right to point out the potential dangers of what his son is doing.
Here's another thing to think about, in Xander's defense. When Buffy was dating Angel, Xander hadn't yet embodied any sort of double standard; he was merely concerned for his friend, and yes, probably jealous that she wasn't with him. Generally speaking though, the issues that he voiced against Angel were not incorrect. Yet in spite of this, Buffy loved him, and he loved her. Is it not possible that when Xander considered dating this former demon of 1100 years, that the notion of Bangel was a bit of a factor? Perhaps on some level he was inspired by it; he'd learned that a demon can love a human and vice versa, and there was a bonus here--Anya was physically human so they wouldn't have the same ropes to negotiate that Buffy and Angel had. And to keep this consistent, when Spike has his soul in season 7, Xander's disapproval is always based on fairly rational concern, in the same way it was with Angel in earlier seasons.
So to sum up: yes he's hypocritical. Dating Anya is not so hugely different from dating a current vampire for all of the reasons I've listed above, (and in some ways potentially worse as I've stated--when it came to this particular former demon). However, Xander is human. He loves Buffy and is not unreasonable in his critique of either vampire. If anything, it bothers me far more that he's inconsistent in how he deals with both vampires, himself. ("We should totally stake them, or they need to stay away from us--but they need to save us, first.") But that's a topic for another thread!