I've only been called a gaijin by children, and they all seemed to have been more surprised by seeing me like, "wow, you're a foreigner!" and not really leaning towards having any negative or positive opinion.

I voted "You are not an ethnic Japanese" which seems most correct for me personally. I'm sure a lot of people have been called "gaijin" in a rude way, unfortunatley.