NTA. He might not want to be a father to this baby, but his parents do want to be grandparents and that's their decision. You are tied to him and his family by virtue of the child you're having who HIS parents have expressed an interest in having a relationship with, as well the fact that you on your own have a positive relationship with his parents. He factors very little into this decision, and it's really weird and self-centered of him to think that he's the one who's important here when he's not. You and his parents are independent of him and are free to have whatever relationship you choose. He just doesn't like that you're going to be around all the time and that he can't put you and your child out of sight and out of mind like he wants.