Yes, your ex is trying to continue controlling and abusing you. You are not stupid, except to the extent that you entertain any thought that he is doing anything else on an intellectual level. Emotionally, you don't want to believe this; that's normal, and you shouldn't blame yourself, because you are a victim. But intellectually, you must disabuse yourself of anything else.

How do you stop him getting into your head? By never talking to him again. You need, not should, need to go full NC with him. As long as he has a way to communicate with you, he has a direct line to your head, and you will never be free.

I've been on the receiving side of emotional abuse; you don't owe this guy a damn thing. Do not call him; just block all forms of communication. Everything. Leave not a single way for him to contact you that it is possible to block. Perhaps send him an email telling him what you did, that you will never communicate with him again, and that any attempt to contact you will result in a restraining order. You are not doing this because you owe him anything; you are doing this for your own peace of mind and because there is a very slim chance that it might head off the need for a restraining order. If he keeps trying to contact you, get a restraining order. Change your number. Get new email addresses. Move back home for awhile. Whatever it takes.

I would suggest telling someone you know and love the full situation, since I would not be surprised if you haven't (I certainly didn't). They can help reassure you about the rightness of the decision, support you in it, and be a source of strength when you are wavering. If not this new guy, then perhaps a family member. Your ex has likely cut you off from others, but they are almost certainly more ready to renew those connections than you would believe.

Also, this new guy sounds great. You should keep that up.

And seriously, please, if you care about yourself at all, cut all contact with your ex.