Yeah, he should stop having anxiety issues with the power of his penis! Fuck that shit. I know I may not seem like I'm being too kind to it, but I've just had to spend several comforting a guy with anxiety issues about talking to most anyone about anything, and that sort of 'Man up!' thing is a big part of it. So forgive me if I'm not in the best mood for this.
If you want to go out with someone, and they want to go out with you, then it's equally incumbent on both of you to do something about it. Yes, it's frustrating, you're both anxious. Believe me, I've been there. And you don't HAVE to do anything about it. It's totally fine for you to both keep doing what you're doing. But what's really not fair is for you to not do anything, and blame someone ELSE for not doing anything. Men do not magically have immunity to anxiety, and women are not frail creatures who can't do anything on their own.
Anxiety sucks. I know it. But it's not fair to someone that you like, to blame them, but give yourself a pass. It's good you know it's just an excuse, but that doesn't make it fine. And he might have his own excuses. Excuses aren't the same as reasons. Either it's fine for both you and him, or it's not fine for anyone. You don't get to opt out of equality when it's inconvenient for you.
I'm not saying you need to ask him out. You make your choice, whatever it is. You can choose to ask him out, you can choose to wait. You can also choose to be a hypocrite, make a choice, then be pissed off someone else would want to do the same thing. But admit what that is. It's hypocrisy.
Now don't think I'm not sympathetic to your anxiety. I am. Really, I've been there. Anxiety sucks. And it can be painful to deal with. You can choose to try to push through it, and you can choose to bow to it, and neither one of those makes you any less of a person. Some things ARE too hard to deal with, and if that's the case, then fine. I know, I've had stuff too hard to deal with. But it's really not fair to someone else to put extra pressure on them. If you can't deal with it, it's fine to admit you can't deal with it. Some things are too much to deal with.
And I know that can be frustrating. But try not to project that frustration outward. Double-standards don't make the problem go away. It's okay to be frustrated with yourself. You're the only one that you can control. And for what it's worth, I do believe you have the ability to control yourself to ask him out.