You see this feeling you have for him? That's how he felt about you. That's how he thought you felt about him, but you didn't. I'm forgiving to a fault because I grew up in an environment where I saw a lot of good people make really bad mistakes, and learn from them and move on. However I don't think you've actually grasped what it was that you did wrong. You keep saying you made a mistake but you didn't. You did something and felt really bad for it afterwards.

I have absolutely no doubt that you feel bad, and that you're genuinely remorseful and love your boyfriend. But you didn't love him when you did that. That's what he can't get over. You weren't raped, you say that yourself. If you loved him you would have run out of that room because you would have felt raped, because you would have genuinely not wanted to be there. You probably love him now, that's why you wouldn't do it again, but you didn't back then. You thought you could do better. You went out and tried to do better. It's the very definition of not loving someone. Seeing this guy is a reminder that all those years leading up to it were not as special to you as they were to him. He thought he had made a special connection but he didn't.

I don't agree with people saying you should break up, but you need to work on the things you actually have control over and accept them things that happened. You didn't love him when you did what did, then you slept with someone else and then you actually fully fell in love, you finally realise what he had already realised about you. You took something away from the relationship you can never put back, ever. It doesn't mean you can't now build something new.

It's not really about the sex, otherwise we guys would never date a woman who has had a previous boyfriend. It's that he knows you didn't think sex with him was good enough. Stop treating it like a mistake you can fix because it wasn't a mistake, you literally went out and had sex because you didn't think he was good enough. You can't fix that any more than you can stop yesterday from having been yesterday. From what you say you seem to have been doing the right thing. You can work on making him feel special now and it may come with time. But you can't give him back something he never had. You didn't love him when you did that to him, that's what he can't get over. This guy just reminds him of that.