When I was pledging I had a very similar thought to you. I wasn't sure if I fit in with my chapter at all (wide variety of personalities, apathy, etc.). I considered dropping multiple times and stuck through it mainly because I am not a quitter.

When I became a brother I immediately began changing the aspects I didn't like. The house was becoming damaged and neglected, I became House Manager. Weak leadership, I became Vice President. Cliquey brotherhood, I became Brother at Large. Shitty semi-formals, I became Semi-formal chair. Failing pledge program, I became Pledgemaster. I can keep going, but you get the point. Even in a place I didn't feel like I fit in, I made sure the fraternity I joined was up to my standards. I guess this could be seen as selfish because I was influencing my entire chapter to make certain choices I felt was correct. Nonetheless, I was awarded both Brother of the Year and Senior Brother of the Year for my efforts.

Even with this in mind, we did have pledges that have dropped like 80% of the way through pledging and were much happier at their new chapters. No hard feelings. They found their home and felt comfortable. I just cant imagine going through pledging twice. Deep down, all fraternities are a collection of young men looking to pursue a higher education in combination with women, leadership, and brotherhood. I guarantee you there will be a similar issue of "not fitting in" in every chapter. I have made best friends with guys I wouldn't even talk to in real life. Jocks, nerds, musicians, assholes. It is a part of the fraternity experience. You face turmoil, triumph, fun, misery, excitement, and memories that honestly last a lifetime. I can tell you that I my heart was definitely not "invested" in the house I was in until I was able to improve the quality for everyone.

Hope this helps give some perspective.

Basically, you can morph your chapter into what you want it to be or you can join a chapter that is already how you see fit.