No I understand what you're saying and I think it is a stupid point. You're saying just because something doesn't completely eliminate you from the dating pool you can't be annoyed that it is a noticeably disadvantage.

You even said this:

you may not be found especially hot right off the bat

You're admitting that it is a physical feature that will often make dramatically reduce your attractiveness. Yet at the same time say

You're self-conscious about your height for no reason.

You know other than the reason being that it makes you considerably less attractive.

Let me rephrase this in a different light so that you might see it fresh. Imagine if a fat guy was saying he can't get a date and often girls skip over him because he is fat. Sure there are fat guys out there with girlfriends, it isn't like being fat makes you some hideous creature nobody wants to talk to, but you surely won't be a first round draft pick at the bar. The best girls can and will simply match up with fit guys and the fat guy might be able to get a girl to like him by simply having a great personality and being fun. The obvious advice to the fat guy would be "stop being fat", but you can't actually change your height so it's a source of annoyance and bitterness to a lot of short guys that they have to essentially accept the fact they're second string in the dating pool because of their height. You can fix your teeth, get shredded, dress and groom well, but you can't get taller so you might imagine it is a bit annoying to have someone who is 6 foot saying "you're acting like it matters but it doesn't". It'd be like me telling someone who is genuinely ugly that it doesn't matter and confidence is everything. It isn't, sometimes if you're short or ugly you just gotta accept that it's a pretty noticeably handicap, living in denial won't suddenly make you some tucker max style playboy.