It's not that the raids are super hard (they're reasonably difficult), it's that getting a raid going is often much harder. The community produces a lot of negative secondary experiences in the raiding sphere which turns many people off:
... Those are all that come to mind, anyway, from my own off-and-on raiding experiences. The hunt for a decent raid is often harder than the bosses themselves.