I can relate to your son having parents that split up at an early age. In my situation my dad and my stepmom was the good caring parents that raised me well. My mom fought tooth and nail for custody of me, and once she got that she either neglected me or was verbally abvusive and a myriad of other things. Since my mother never listened to me and understood that i wanted to live with my dad and kept up the abvusive behavior i have now completely disowned her and shut her her out of my life.
I guess what i'm trying to say here is find out if your son wants to spend more time with his mother. Even if that means spending less time with him do it for the sake of his mental health.
One final note here i'm not saying any of this to insult you and your wife as parents. I'm just giving my two cents based on what you've written here.