Your father was not wrong. He reacted organically, the way he felt, and he was being supportive. HOWEVER, I've read from some gay people that being dismissive of the coming out isn't the best choice, and your father was somewhat dismissive. It's better to say, "It must've been hard for you to tell me. Thank you for trusting me. I love you. Let me know if there is anything I can do." Acknowledging how huge and scary coming out can be is a good thing.

But your nephew wasn't offended or upset, so no, your father didn't do anything wrong.

Your SIL however is trying to be a fierce mama bear in a situation where it isn't needed. It sounds like she is looking for a fight so she can support her son to the fullest and is being a bit oversensitive. Overall, I would say your SIL is handling the news and reactions badly. She needs a group she can talk with. I'd suggest contacting the local PFLAG.

It is not your place to get in between your father and your SIL. If anything, let your brother handle it. You just need to keep supporting your nephew.