It's normal for people to say awful things to us because that's what they see or believe. If you think your doing the right thing then continue what you're doing.
According to Bangambiki Habyarimana (The Great Pearl of Wisdom) “Don't go into the business of pleasing people. You can't please everybody. Simply do your best at what you do”