If I was watching the whole series for the first time today, I would probably say the Gentlemen.
After the master, who I first saw when I was ten-ish, I hadn't gotten truly scared until Hush. After that, the other monsters seemed friendly. The thought of someone floating behind you as you run, brrr. And the silence...