Let me tell you about Calvin (name changed to protect the minor). Calvin is our family friend's son. His mom is a child psychologist and his dad is an artist. When Calvin was born the doctors told his mom her "eggs were old" so to expect some mental disorders.
As Calvin got older he wasn't able to make friends. To put it simply: he's just fucking weird. His dad is really obsessed with historic warfare and used that as a bonding point with Calvin. He enlisted him in Boy Scouts (and now Eagle Scouts) so he could learn "boy stuff". But at every family gathering my cousins and I just avoided him. He is a year younger than my youngest cousin (who is now 19), but my younger cousin just couldn't relate to him. One time at a barbecue, my cousin Adam went up to my aunt and asked to go home. When she asked what was wrong he said, "Calvin said he wants to tie me up. I want to go home." Calvin was only 5.
Whenever we are around him he plays imagination games where he's always shooting or fighting someone. He will walk into a room, pretend to shoot it up, and walk out. Three years ago at Christmas (he was 15) he brought wax and made wax figures that were spear fighting. His mom, the child psychologist, has ignored all this behavior and just said "he's different."
Let's get to this past thanksgiving. Calvin is now 17 (almost 18). He spent the evening burning carrots over a candle and the eating them. He took pictures of the steak knives my cousin's wife set out so he could research them later. But the nail in the coffin for me was when he started asking me weird questions. Example: "if you were married to a guy who was a war veteran but was on the brink of a psychotic breakdown where he was a step away from murdering you how would you feel?" Verbatim. I started to get uncomfortable so I turned to talk to my grandma. But when he got up to walk away I noticed he had a very visible erection.
That's it. I've known this kid his entire life but I told my family I will no longer be going to family functions if he is there. He is associating pretty graphic violence in its sexual arousal and I do not feel safe around him. He is much taller and stronger than I am and I do not feel safe around him. My aunt then brought up a time a few months prior where they had a bingo night at Calvins parents house and when one of the women was using the restroom she saw Calvin peeking in the window.
Frankly, in my eyes, it is not a matter of if he murders or violently rapes someone, it's a matter of when. And his mom refuses to acknowledge that his issue is that severe. They are no longer welcome at family gatherings until there is marked improvement.
Your sister doesn't want to admit her son is a time bomb, but he is. And you know what, maybe he never acts out, but there's always that possibility. From the other side: he has all these thoughts and feelings and emotions of wanting to do these actions that he knows are socially unacceptable or wrong (which is why he's unable to relate to other kids) but he doesn't know how to control them or why they're wrong or what to do. You need to set up boundaries with your sister and stick to them.
"Until you seek professional help for you son we can not be a part of your life." And that's it. Don't go to family functions where they are present, don't invite them to family functions, etc. It's difficult to cut off a family member like that, but you're not doing it without purpose. Your daughters health and happiness is your number one priority so you need to do everything to ensure that she is safe and happy. It will hurt you and your sister and everyone else involved, and it will be so so hard. But her son needs help.