If he can get hard but loses it, it's psychological and that can be numerous things.
Is he ashamed of sex? Tell him to be honest with himself and think deep down if he believes he might be taking advantage of you?
Repression? He wants you to treat you like a dirty whore but simultaneously thinks it is wrong and consequently have hang-ups about it.
Does he watch porn and jerk it? Both visual and physical stimulation can go on overload and he needs to reset; his brain has to mellow out from all the hardcore fuckin' and his dick needs time away from his "death grip."
Condoms can definitely be a problem. They might be too tight, or it might straight up be a turn off for him.
I guess it's hard for me to comprehend as we're taught to liken sexual attraction with how hard someone is, are you still sexually attracted to the person when you can't sustain your erection?
Okay; you've certainly experienced fear of heights, right? Or nervousness before an exam? Now imagine enjoying yourself in the sun and drinking a good wine and your vagina's getting a real good time with his tongue and then FUCK you just remembered the exam you have tomorrow and you haven't studied at all and it's way too late to start now as it's already past midnight. Now try to stay horny.
Might be a bit like that for him, and that's only for the anxiety for the ED, this is not including the possibility that he might think he's doing something morally wrong if he did finally manage to put it inside you. You know, "keep it in your pants, ugh men are such pigs, respect women."
Don't withdraw, he needs your touch when it happens or he'll most likely end up with more anxiety around it. Cuddle and snuggle and do all that cutesey stuff. Put on a movie and just act like teenagers.
TL;DR: Therapy.