To be fair... Ned Stark wanted to back Stannis and Varys viewed Joffrey as the lesser of two evils because of his hatred for the lord of light, as he mentioned in the war against Stannis " The dark arts have provided Lord Stannis with his armies and paved his path to our door. For a man in service to such powers to sit on the Iron Throne, I can think of nothing worse." What kind of made him special in the show is that he's benevolent when it came to the big picture, but indifferent on small details. Kind of the opposite of Ned Stark, who fixated on every little thing but missed the big picture. He also kept his word and told Dany she was making a mistake before betraying her, and she ignored his advice.