I just want to make sure this is clear: this is an incredibly awful, traumatising thing that happen to your nephew. That it happened at a family gathering by family members, and that the incident was dismissed by some of the parents (!!!!!!) is even worse.

The most important thing is not your guilt or teaching your daughter a lesson or assuaging your brother. The most important thing is for your nephew to heal.

In addition to talking to your daughter about bullying and consent (which are so so important), make sure she understands that an apology isn't enough to fix this. That she may not have intended anything other than sucking up to the big kids but that the effect was devastating. Saying I'm sorry does not equal the violence and violation of trust inflicted on this poor child. This is going to take time and ongoing proof that both your daughter and you are trustworthy.

Talk to your brother. Make sure he knows you have his back and ask him what you and your daughter can do for your nephew. That might mean no contact for a while. It might mean cutting off the cousins who didn't think this was a big deal. This might mean your extended family finding out that your daughter participated in a group sexual assault--yes, she did, and if you want any chance of healing you cannot ask for this matter to be hushed up and forgotten. This might mean going to family therapy together. Whatever your nephew needs, regardless of whether it makes you feel ashamed, you should do.

Also, seriously, cut those cousins from your life and never let your daughter around them again. Not only was this incident horrifically repugnant, but next time your daughter could be the target.