Yep it's definitely a matter of timing. Here's a blow by blow of how it ttypically plays out for us:

1) We all gather up by the "crystal room/hallway" exit to the left. When everyone is ready, we all exit and head straight to the middle ledge in front of crystal room.

2) Everyone is now in position along the ledge. Swordbearer door is opening, thralls are pouring out. He's about to step out.

3) As he's stepping out, I'm shooting 2 rockets (dumbfire, not tracked) while jumping down toward the middle of the arena. 1 rocket just to the right or left of the little ledge blocking his door, depending on which way he goes, and another right after in the same area, hitting him squarely if possible. The other guys are still up on the ledge sniping him and hitting him with primaries. By the time my rockets are cleared I'm on the ground, now in range of boomers and Crota. I'm jumping while moving forward toward the swordbearer. Boomer shots are landing all over behind me, and Crota's shots are going over my head. Thralls are taking swings and missing because I'm moving and jumping.

4) USUALLY the swordbearer is dead by the time my second rocket hits near him, thanks to all the snipes and primary fire from the other guys. I continue pushing toward him, continuously jumping to make boomer shots and crota's shots miss. When I get to him I scoop up his sword and continue on with the fight.

If the swordbearer is NOT dead by 4), it means he went way far right or left. If the previous steps were followed, he's probably still down to about half-ish health thanks to the initial rockets from me and snipes from the non-swordbearers. This means I continue toward him on the right or left, jumping to avoid shots. Quickly dump another 2 rockets into him, stick him with a nade, and then he's most likely dead. From there you continue as normal with the rest of the fight.