Put your foot down and make him do it. This is not your child and he doesn't get to pawn logistics off on you when he made the decision to stay there.

Also.. speaking from experience it is super difficult to have any identity other than caretaker/mom when it's just you. I ran into this with my ex and we lived in the same town. I got no help. I love my kids but I was ready to find the nearest circus. So her complaints aren't totally unfounded. I tend to sympathize with mom on this because barring high conflict crazies she's been doing all of that alone. If he's willing to dump it on you I highly doubt he's been helping her past child support.

And at the risk of sounding totally callous.. the kid didn't ask to be put in this situation. If you guys plan to stay together then could moving closer to his son be an option? As well as finding a situation where you could afford to support him? This is the reality of being a parent and while it isn't your responsibility it is his. It affects you should you stay together. You may as well find a way to make it.