You have to actually care for it to be an insult. 90% of the time when somebody says that to me what they don't realize is their opinions are fly crap on a dog turd to me (really meaningless) and that I likely see them as more of the same anyhow so I can't be bothered to care what the have to say.
Comments