People have been talking way before you were born, and they will continue talking till the day you die. They will forever have opinions about you and the things that you do, no matter how hard you may try. And do you know what the most interesting point is? They will continue to talk even way after you death. Everybody has the right to their opinion, isn’t that what freedom of speech is about.
Let them hold on to whatever they may be mumbling in the dark corners of their offices, and street corners. Just remember that the most important thing during your existence is you, therefore you should concentrate on yourself, and for a change ignore the negative.