The people to worry about are not those who openly disagree with you, but those who disagree with you who are too cowardly to let you know.
And then they move silently underground. Always do what is right even if it is unpopular.