One of Joe Silva's only media interviews from a few years ago explains why some fighters might dislike him (link):

The thing a lot of fans know about Silva – or think they do, anyway – is that he’s, in the words of White “a mean little f—er.” That just happens to be Silva’s least favorite description, which might be why White loves to needle him with it.

“I really don’t consider myself a mean person at all, but I am very honest,” Silva said. “I won’t lie, and I put things very straight. And fighters, they have a lot of buffers around them, sort of ‘atta boy’ guys. They’re not used to someone telling them, ‘No, this is how it is.'”

That’s especially true when it comes to contract renegotiations and bout offers. Fighters, like most people, aren’t terribly fond of having a dollar value put on their worth. They also don’t always like Silva’s ideas for whom they should face next. When they hear his no-nonsense explanation, which some managers say can be Vulcan-like in its unvarnished appeal to cold, hard reason, some might understandably come away feeling like he’s a little bit of jerk.

But then, as Silva sees it, that’s kind of the way it has to be when you’re offering one person a chance to make money fighting someone else in a cage.

“If everybody only took fights that they thought they had a really good chance of winning, nobody would ever fight,” Silva said. “When you look at the odds, they almost always favor someone, even if they end up being wrong. But if you’re in the UFC, you’re here to fight who I have. If you’re going, ‘I don’t think I can beat that guy,’ then maybe you should be doing something else.”