If there are multiple victims of sexual assault and if you have credible evidence of this, then you should encourage the victims to go to the police. This isn't a matter which should be dealt with only through church discipline. In reading between the lines, it doesn't seem like your church would be the most supportive environment for the victims, so I can understand why they wouldn't want to "make a big deal of it." This is where you are needed to advocate for them and mediate between the pastor's family. The brother's homosexuality, though sin, is irrelevant and secondary to his alleged crimes and I would ignore that aspect the situation for now.

I'm in the same camp with other commenters that there's no reason the brother's actions should affect your relationship with your girlfriend. However, rightly or wrongly this has soured you on the relationship. Rather than unilaterally making the decision to end it, talk this through with her.