None of the 6 major ethnic groups really had the power and numbers to exterminate the others and permanently suppress political opposition, at least not in the long term. Remember that most of the region's ethnicities hated each other and thought the former Yugoslavia belonged solely to them, the Serbs just happened to come into the most power the fastest. The Serbs might have killed tens of thousands more Croats and Bosniaks given half a chance, but eventually there would be a united backlash. It probably would have succeeded, then Yugoslavia would either separate into 5 or 6 countries as in OTL, but over a longer period and with a lot more bloodshed, or devolve into ethnic warlord feudal anarchy, as in OTL Somalia. The central government(s) regaining most control would likely happen far sooner than it has in Somalia, given that there wouldn't really be an element of religious extremism involved, but with the region's Muslims and Christians under serious ethnic pressure over a longer period I could certainly see that changing.

I would guess that there would be at least one Communist dictatorship remaining without western influence pushing all sides toward liberal democracy, most likely Serbia, but I can't say that for sure given the trajectory Eastern Europe's other former Communist states took on their own. The people of Yugoslavia were generally more sympathetic to Socialism because their ruling class and standards of living were better than those in Communist states that were closer to the USSR. But that was less true immediately before the war, economic insecurity caused by the collapse of the Eastern Bloc was what laid bare the ethnic tensions, so it's hard to say whether someone would remain Communist and who it would be, but it's certainly a possibility that would exist that really didn't in OTL.