If you want to avoid inbreeding (or at least keep it to cousins) and leave room for accidental deaths of people who are fertile or haven’t reproduced then you just increase the number of people. Exactly how many depends on how much you want to guarantee survival. 5,000 would almost certainly be enough provided there isn’t a catastrophic disaster or plague or something of that nature. I don’t think there’s a point where it goes from “not okay” to “okay”, though. It just gets less risky for each person you add.
For biological conservationists, the minimum number of individuals required to maintain a species is called the minimum viable population (MVP). The MVP for most species in the wild is around 5,000 without human intervention. If you include human intervention, a study on pandas showed that with 50 pandas, there was a 70% probability of survival.