Let what others say of you be their own concern; whatever it is, they will say it in any case [Loeb Classical Library translation by C.W. Keyes].
(Latin: quid de te alii loquantur, ipsi videant, sed loquentur tamen; my translation: "What others may say about you, let them see to it, and indeed they will still speak.")
Cicero, De Re Publica, "Scipio's Dream," Book VI.23.
Like all other aphorisms, you must apply this one in its practical context. In many contexts, what others will say should not be high on your list of concerns if you are doing the right thing.
(Image in public domain)