Ah, this problem. Have had my fair share of headaches from this myself. All I ever wanted was to finish the Origins Easter egg on call fo duty zombies, but anywhere from 10 minutes to 2 hours in, I would disconnect randomly (on the 360 side, not the One side) and lose all my progress. It took the living happiness out of me for a good while.

I ended up researching a ton about this. Troubleshot my Xbox, my router, my modem and everything in between. In my case, it turned out to be my router. Even though this problem only started after I got an Xbox One X, I do believe it is mostly the router and not my Xbox. My theory is that there was some kind of compatibility issue between my old router and my Xbox. I switched the router out and no more disconnecting.