NTA for being upset over this. If you were to hastily jump to conclusions or lash out i would say YTA, but it sounds like you have not acted rashly yet, which is good.

It does seem a little weird that for some reason he can't have you and the ex-wife over at the same time, considering it sounds like they are on decent terms and presumably you don't have any beef with her. I would definitely press for a more in-depth answer to that one in particular because if there isn't any hostilities i don't see why he had to change plans. And if i wasn't given a particularly good reason to why plans were suddenly and so casually changed over just that i would also be angry. But you want more info on the situation before you make a more permanent decision.