In our current political status, as much as it pains me to say this, we need some sort of power to hold people to a certain of level of freedom which should act on said actions that violate that freedom (i.e. the scenario you suggest.)

Keeping that power from becoming abusive or permanent is another problem altogether. Allow me to use an example: When cities were on the rise in America and we had a large urban boom of populations, food vendors suddenly found themselves needing to meet a demand they did not have a supply for and some shady actions were taken. There were many cases where spoiled meat was sold as fresh, and the go-to move was to mix saw dust into basically anything that you could mix saw dust into to make it seem like there was more of it. Thus, the FDA was created with the noble intention of preventing these things. The FDA did a lot of good in its heyday but has since run off the rails with frivolous rules that hurt small businesses and don't even get me started on their erroneous food pyramid with bread at the bottom that helped to turn this nation into one of the unhealthiest on earth.

My point is: that although the creation of the FDA was a violation of current libertarian principles, it served a need at the time and now that it is largely obsolete in many ways, we're having a hell of a time getting rid of it. Is our modern problem worth the problems it solved when it was relevant? Maybe. Maybe not. So too, with your scenario. Currently, in a society which can vote to violate certain protections and then secede, a big bad federal government is needed to be able to stop that from happening. Will this cause problems in a future society when the idea of coercive force via democracy has long been taboo? Probably. Will the costs offset the gains currently? I don't know. But may our great-grandchildren have mercy on our souls.