Yes, but you will notice that there are members of both the groups you compare Paganism to that say thoe words should not be used at all, by anyone, and others who want to "take them back." It is a politically sensitve issue. There is also the problem of how can we say we're all equals if one group can use a term, but other groups can't?
Not saying don't use it, necessarily, I use it. I used it in an above post. I just want to make sure everyone is as careful with the context as they would be with, as you so astutely point out, the N word, or "fag".
And it referred in the past to anyone who believed in "primitive, country" things, such as multiple Gods, the power of ancestors, the cycle of nature, reincarnation, magic, and animism. It did not include, for example, Jews, Mithrans, and later, Muslims. People had their own derogatory terms for these groups.