The question was pretty well answered on the original "Star Trek" series. Kirk was dropped, alone and unarmed, on a planet where he had to fight a Gorn (no, not a cave man, but a brutish, savage reptilian being.) When Kirk tried to fight the Gorn directly, its superior strength made all of his hand to hand combat training useless. It wasn't until he used his superior intellect to create an explosive device that he was able to win. The parallels aren't exactly the same, because the Gorn was from a space fairing race and, supposedly, as scientifically advanced as humans, but instead of relying on intellect, it chose to fight on a more primitive level, going so far as to arm itself with a stone knife.
As with Kirk and the Gorn, in a physical confrontation, the brute, or caveman, would win. What made Kirk, the astronaut, the ultimate winner, was that he was able to break away and postpone a final conflict until he could use his superior intellect to create affective weapons that would allow him to overcome the caveman's superior strength.
So my money's on the spaceman, just as long as he's at least as smart as Captain Kirk.