I believe the problem was, in part, fact that the Ferengi did not seem to be such an existential threat. This was achieved by the Romulans and the Cardassians, but the approach was very different. One part of Star Trek that people for some reason try to edge around are the action-adventure aspects that are pretty important. There are phasers mounted on the Enterprise for a reason. The Romulans and Cardassians gave the opportunity to use both brains and brawn.

The Ferengi... did not, at least, not in the same way. Roddenberry was very set in his political and sociological views by that point, and had become much more devoted to pushing them into Star Trek whether or not they were actually a boon to the franchise. The "horror" they supposedly espoused was involved in their capitalistic and sexist approach. He seems to have intended the audience to shudder in horror at this ideological thought as well. Even disagreement, however, does not equate horror, and as such they seemed like (and pretty much were) hissing, funny-headed weirdos with whip-guns for some reason.

Suffice it to say, that didn't work out the way he had hoped.