I started as a BD as well, and after reading so much hate (maybe dislike is a better word?) toward them in PvE in later content, I rolled a summoner for PvE, which eventually became my main, due to my horrific ping and summoner's low reliance on it.

Basically, a couple things that I read, and what I ultimately ended up agreeing with:

Destroyers can do just about everything a BD can in PvE, and do it better. Their grab is longer and less awkward. The BD grab is awkward for the rest of the group, in that the hitbox is odd and can mess with DPS if people aren't used to it, and sometimes even when they are used to it.

Destroyers have better sustained DPS. BDs do have some decent burst (which makes them good in PvP), but rely a lot on draw stance, and, as other people have pointed out, CCing requires you to break stance.

For me, there were a couple other annoyances when it came to maintaining stance as well. We aren't yet at high level content, and pretty much any pug will make annoying mistakes that mess with your ability to maintain stance (running around needlessly while grabbing, knocking enemies up in the air, etc.) Any enemy that teleports around also got annoying for me. The sisters in BSH are a great example, but pretty much any sin-like enemy messed with my ability to maintain stance because I had to chase them around. These things didn't make sustained DPS impossible, just annoying.

In the end, Summoner made more sense to me. I have a huge amount of utility in groups, being able to heal the group or give a chi burst when I have a really good group that knows what they are doing, CCing, dropping my cat when I'm in a group with no tank or a tank that isn't competent (again, I Pug a lot). Their solo ability is crazy as well. This really needs no explanation.

In the end, I wish I had rolled a destroyer first, now that I have more knowledge of the game, and don't really care to roll another melee class right now. They are just...better than BDs, and will be for the foreseeable future. BDs won't be as disliked in groups as sins are right now (whether that dislike is warranted or not), but will be in the "middle of the pack" for some time to come. For this reason, although you may not have people drop from your groups or turn you down, your group would much rather have one of a number of other classes in your spot than you (again, warranted or not).