Toxic, I would see couples therapy if you want the relationship really bad for other good reasons. Otherwise just cut your loses and move on. He is 18% of your age older than you and a has at least this one clear identifiable factor of narcissism based on how you explained this.

I have listened to 100% of anything any of my EXs have told me and I have never been judged or have begun to think upon on judging an EX for her sexual actions before my relationship with them. Judge them for their sexual actions after the relationship? DAMN straight I held each of them accountable to who they said they wanted to be going forward. But I don't judge them, hate them or anything negative just because I can see what's in their shadows. You can see his shadows right now and there is not reason to hate him but attempt to help or leave him. Those are really your only two options. A change must happen.

The reverse toxic situation would be if you told him you weren't sure about him because he was able to have unemotional sex.

He needs some goals and a path forward to not worry about if he exceeds your previous lovers because that is the NUMBER ONE thing that will eventually destroy any chance at a future together for you two. I just went through a big change I never wanted to go through but I think I am learning change is the easiest thing to go through until you stay still too long. Grow together or apart but don't stop improving the happiness in your own lives'.