I got into Karate at a very young age. I got the opportunity to train with people both smaller and significantly bigger than me. Once I got my black belt, I noticed that no matter how much bigger you are compared to me, if you don't have the same training as I do, I can beat you up. No matter what.
Why I'm saying is that since I got my black belt, I never considered height to be any more advantageous to a person than for a woman to find you attractive. I'm 5 foot 7, and I've overpowered 6 foot guys, considering that they don't have a black belt too.
If you try martial arts training, where you get to train with people bigger than you and you overcome them, you won't feel like a pushover and won't even think yourself small.
Karate wasn't my key to self confidence, but it definitely stopped me from being insecure about small I am compared to bigger people.