You can tell Xander and Giles were never going to be really close, but I always kinda figured it was cos they wanted Giles to be the father figure to Buffy and not anyone else.
*Oh, I remember one moment between them! It was an episode about Oz being a werewolf, and Xander said a joke with the moon and Giles thought it was super funny. Otherwise...I don't remember any special connection between those two!*