I have a crippling self-esteem issue from me having achieved nothing at 23 years old(on top of the self-esteem issues I already had in high-school)

You're in college 3-5 years later than the majority of kids. That's not a problem unless you make it a problem. Don't waste the opportunity you have now because you're looking ahead. You've got to enjoy the process and appreciate everyone does things on their own time.

Plus, you're going to be what, 26-27 when you leave college? That's nothing. You're the standard age of the classic college late bloomer. I wouldn't say you're a 'weird' age at all, a few people on my undergrad were your age, and during my post-grad year it seemed like every 3rd coursemate was 25-27. We all still went out (although the 18-20 y/o's do start looking very young).

If you don't like partying because it's not your thing then fine - that's always tough in college. But if it's an age thing, I go out most Fridays / Saturdays in London and feel relatively young out in bars and clubs. In fact, the night life now is better than ever.