ESH

But still, remember, its not always about "his rights", or "Your rights" Its also the child. I'm sure the child would like a father figure.

Of course, it depends on the situation, but to me this situation looks like the common custody battle, and that atleast shows that he has interest in the child now.
It scares me how ignorant people like you and him are, I'm sorry, but the way you write it, it makes you TA on that part.
" I'm now refusing to ever let him see my son "
He's 100% also TA, but this fight should not only be about you and him. Also about your son. If he is generally a normal/good man, wanting a relationship with his son, the best thing is to slowly give him it, for the son. Even if he might've been a dick in the past, for the son to have his father, well, he'll be glad for it.

This what I hate about custody battles. It's always about the parents, never the child.