A lot of horrific "advice" from Reddit in here, as usual.

If this guy really is the "in all other aspects perfect, most amazing partner" - just please fucking talk (more) to him. Don't let your imagination go out of control and certainly don't let Reddit's collective imagination (there are few darker places) in on the action.

Further genuine discussion with him will reveal if this is just some weird idea he's got and never had properly challenged in which case you should try to change that, and then yeah if he still just seems like a lost case and it's an understandable dealbreaker, move on from him.

Would really suck to dump this guy (as described) if it turns out he'll change his mind on this one thing (which - and this is just my opinion - hardly even matters, what's he gonna do about it? And he said he thinks higher of you) though.