Gunn would beat out Xander, Riley would beat Wesley. Then Riley probably beats Gunn. (As much I hate to let Riley win at anything, the "no weapons" limitation really informs my response here.)
Riley is the only one of these four with any kind of formal hand-to-hand combat training, which gives him the edge in this particular scenario. BTVS showed him patrolling with Buffy quite frequently and in many of those scenes he was more than able to hold his own, sans weapons.
Gunn is blatantly referred to as "muscle" on AI's team. He's good with a weapon and, based on his earliest appearances, quite innovative (e.g., tricking up his pickup). But again, this is a weapons-free arena, and that would put him at a disadvantage.
Wesley's strength in the battlefield has always been as a marksman, but he has never exhibited any real abilities at actual hand-to-hand combat.
The point of Xander's character is sort of that he flips male stereotypes around, and provides the sort of emotional support that is traditionally attributed to female characters. His role was never muscle, always heart (although if you want to get really literal here I suppose the heart is technically a muscular organ...). Yeah, he's staked a vampire or two in his time so it's not like he's totally inept on the battlefield, but I don't think he could stand up against the other guys.
