Something very similar happened to me before, I paid extra to sit next to my gf (who's a nervous flyer). Beside her is me and one other guy. The man's wife comes down with a ~4 year old and says to my gf "can we please sit together, with the kid an all.....?" My gf agrees and is as helpful as possible as she's too polite to say no, but I can tell she is a bit upset about it. So I move across to the other side of the aisle and gf moves all the way to the back.

Next thing, plane takes off, the couple send their child up the aisle to where it turns out the grandparents are sitting near the front, and the woman ends up lying down across her two seats with her head on husband's lap and sleeps for the entirety of the flight! In the seats I paid extra for! Child nowhere to be seen until we're back on the ground.

So yeah NTA, if it's important for you to sit with someone, you gotta pay for the privilege. That's how it works.