Your situation is exactly why annulments exist- if he once claimed he was comfortable having children, and now that you're married he has declined interest: that's a form of fraud because he misrepresented himself.

Do not believe a word he says about his ex stripping him of parental rights for tax reasons- rubbish and not how the legal system works. Within family court there is an expression that dictates most matters of judicial decision making: "best interests of the child." And clearly it wasn't in the "best interests of the children" to see their father- which is highly unusual and implies there was abuse, or he chose not to see his children. Family court consistently tries to find ways to keep parents in the lives of children- like foster care: the goal is always reunification when possible and "best interests of the child." It doesn't sound like it's his ex that's a POS- it sounds like he is.

You have experiences that makes you vulnerable to manipulation, that's why he married you. That's why you stay- you believe lies without questioning it.