I’m not trying to be mean because you are clearly dealing with a lot but your solution is a terrible one.

If you divorce your children will be taken away. So, what happens if you’re caught cheating? If his family is truly the way you describe them, they will 100% catch you.

Consult a lawyer and tell them everything. Your specific concerns about his family. See what they think about getting you protection and distance from his family.

Please please please, get yourself out of this marriage. For your sake and your children’s. You deserve so much more than sneaking around to get a tiny bit of fulfillment, that will probably just end up making you much more sad than when you started.