Maybe I can provide a more meaningful perspective than a lot of the other replies you've gotten.

For a lot of my life, I have been bullied and generally disrespected (sometimes to my face). It still happens, and as a result, I am not always keen on socializing or talking to strangers. It's kind of funny, since I frequently associated with the jocks, frat stars, and successful faces of high school/college (in addition to intellectual and nerdier pursuits), though that is not to say that I put down others.

Nevertheless, I think the general mistreatment comes from my relatively short stature (almost 5'5"), obese figure (following the end of my college sports career), and mild-mannered disposition (except when challenged). In other ways, I was and still am lucky. I can be very funny; I am competent; I am charming when needed. Lots of success in and out of school also means that I have had many opportunities to feel confident and successful. Interpersonally, I can count my blessings in being able to understand people fairly easily and having strong success in academics and work. I have also been in serious relationships since high school, and I am going to be marrying the girl I am currently with. No bullshit here: I got lucky. I come from an Asian background, and I consider myself well-versed in and strongly influenced by this background. Generally, this isn't looked upon all that favorably in wider American society either. But what I did was this.

I learned 4 languages because I loved learning languages. I played varsity college sports because I loved playing that sport. I volunteered in the community around my college to help those who would have otherwise had no help because I wanted to make an impact on peoples' lives. I'm attending medical school now to continue on that mission. Regardless of who was by my side (both fun times and not like when my start-up in college died a slow death), none of these things changed. I realize that you did things for "self-improvement" at least in part to find greater romantic success; for me, that's never been the case, but I'm not telling you have to do things my way. I just think that sometimes these things line up better for some than others. Luck is always a factor. To help the odd though, I date offline only. I try to stick with those that would appreciate my talents. It's mostly worked for me socially, and I am more than happy that it worked out for me professionally/academically too.

These lines can be orthogonal; they might not be. Don't invent deficiencies and reasons for self-improvement unless you want to improve. We all have deficiencies that can't be changed. Do it for you (or do it for the girls), but be able to dissociate the two lines. Try to stick to groups that are more open to you being you, and I think that's really the best you can do. It also never hurts to take a break if you feel like it's all just too much.