You have been together for two years, which is the time it takes for infatuation to drop off. He might not even be poly oriented. This could be nothing else than the first step towards replacing you. To be poly he needs to manage both his relationships in a good way and it doesn't look like he has the integrity, will or skills to do that. In your shoes I would focus on becoming independent so that you have the possibility to move on from him when the time is ready.

Are you financially or otherwise dependent on him for housing or something else? Would you be able to leave if his abusiveness escalated or this arrangement gets toxic? Do you have a job? Can you support yourself and the kids that belong to you? Do you have a support network? These are the questions you need to consider here.