I understand what you're saying completely. I mean, I'm a guy, so I think it's slightly different just based on how aggressive men are towards attractive women versus how aggressive women are towards attractive men. But ugly duckling syndrome is definitely a thing; it totally messes with your self-esteem.
Really, where it all started was the fact that I wet the bed... By the time I was 5, I knew most kids weren't still wearing diapers to bed... By 7 or 8 I knew something was wrong, and it continued on until I was 16. Beyond that, when I was 9, I was recruited into a dance program through my school and wound up getting really serious about ballet (I also grew up playing hockey and baseball), which was something I continued with until I was 14. I hit my awkward stage right on time around 12/13, and developed pretty terrible acne on my face, chest and back. I think I really hit rock bottom when I was 14 because all of my peers were convinced that I was gay (which I'm not), I was still wetting the bed at an absurd age, and while I'd been told throughout my young childhood that I was attractive, the acne seemed to take that all away from me.
After a little while, I started seeing a dermatologist to clear up my skin, as well as a urologist for hormone therapy to stop the bedwetting, and I quit ballet because I had to choose between taking hockey seriously or pursuing ballet as a full time commitment. By my sophomore year of high school, all of my issues had been more or less cleared up -- my skin was clear, the bedwetting had stopped, and I was a three-sport varsity athlete for my high school (picked up football freshman year). All of a sudden, I was in everyone's good graces and I was being regularly noticed by girls, even in grades above mine. But it seemed like too many people had known me at my worst time, so the fact that all the background issues were no longer actively happening didn't do much to help my self-esteem because I still felt the weight of all of the emotional baggage I had piled on so heavily in those few years.
Fast forward to college, and I'm playing hockey for my school, at a place where nobody knows who I am except for two other kids from my high school who attended the same college, one of which was a close childhood friend while the other was more of a casual acquaintance. I hit my fucking stride harder than I've ever hit any stride before... But soon enough I realized that people were treating me totally differently. I'm open to making friends with literally anybody, but it seemed like some people who I approached were put off by me because they assumed that I never knew what it was like to struggle or wish that I had things that I didn't have. I already looked like someone who had it all.
To this day, that kind of continues. I meet people in passing, and I know what assumptions they're making. That I've never struggled, that I've always had it all, and that I'm just a naturally blessed person with good looks and a good head on my shoulders. But every fucking day I carry the weight of all of the previous baggage with me -- I still have trouble feeling confident, and I'm often down on myself and worried about what others think. I guess I just got to a point at 14 that has so far been impossible to totally recover from.
Anyway, I'm married now and I have a son, so the things that are on my mind are completely different now than they were even just a few years ago. But man, that shit really fucked with how I see myself and where I think I fit into the world.