This is purely speculation on my part, since it was never fully explained in the show.
Perhaps a vampire cannot see his own reflection because the demon cannot bear to see itself. Typically, vampires do not self-reflect or feel guilty about the things they do - therefore, Whedon's vampires not being able to see their reflection is a really literal metaphor. So going off of that theory, if a vampire is recorded via a camcorder/camera, it works because it's not a mirror they're looking into. And furthermore, all the "mistakes" where Spike and others are seen in little reflections here and there are not actually mistakes at all, because they were not looking into the mirror when their reflection appeared.
Of course, my theory stands on shaky legs, and goes out the window when thinking of episodes like "Reptile Boy," where Willow was looking over Angel's shoulder and studying the way he had no reflection, right before asking him how he shaves. Angel was not looking into this reflection, so by my theory there's no reason why he shouldn't have shown up.
So yes, these are probably just mistakes. But I'm gonna rationalize them anyway! 😛