It sounds like a tough time but in my opinion, stepping away is not an option. Doing that would send your son the message that his father only loves him under certain conditions. It’s very damaging to children when parents withdraw themselves due to personal frustration with the child.

Could you consider setting house rules and boundaries while your son is with you? Or asking him what he’d like to do for the weekend and respecting that (even if it means he wants to be on his own most of the time, in your house). Just allowing him the autonomy to distance himself while making yourself available will show him you’re there for him but respect him.