You're not a massive, piece of shit, arsehole, scumbag. Theres no reason to think so given what you've told us.

You should pat yourself on the back for not being a social retard, making an effort is not easy.

You should consider taking it a step further and meeting even more people. Why limit yourself to a friend group that gives you so much anxiety? Don't hate them for it, don't hate your bro for it, and especially don't hate yourself for it. Some people just jive with eachother better than others, thats life.

I know its hard for you to realize, but it actually sounds like you're doing really well. Could you imagine what highschool you would think if he knew you actually went out and made some friends?

Keep it going dude, you're doing great.