It's possible you may not have been making use of the damage reduction passive skills that they were. Some examples being Reimu reducing all spirit damage if she's in the frontline or Mokou having an invisible +30% defense and mind buff if Keine is in the frontline.

The same applies to offense. Tenshi is normally a fairly weak character but if Iku's in the front and you've invested in their synergy skill, she gains an invisible +40% attack increase which makes her significantly stronger and one of the better nature element attackers.

It doesn't necessarily need to be only four characters. Just however many you absolutely need to win the fight. I believe LG used five or six characters in his video for example.