In the short-term, yes, many firms have incentives to act in ways not in their customers' best interests. However, a firm's reputation is enormously valuable (this is why branding exists). Consumers tend not to suffer firms which screw them. In the medium- to long-term it is almost always in a firm's best interest to maintain their reputation and provide good service. For example, if you go out to a restaurant and don't enjoy your meal, the restaurant could require you to pay for it anyways. In the overwhelming majority of cases they don't. Why? It's is absolutely in their short-term economic interest to make you pay for it. They're suffering a direct and immediate loss if they don't. So why don't they? Because it costs them your repeat business and likely also the repeat business of anyone who you relate your poor experience to. It hurts their reputation.

Compare this to the case of a government monopoly in court and police services. Society on the whole has little recourse if they court and police system is failing to reflect the moral norms of the society (think no-knock warrants, the war on drugs, mandatory minimum sentencing, and any of a slew of other ridiculously unpopular developments in the criminal justice system). It's true there is some feedback mechanism through the democratic process, but that feedback mechanism is weaker than the market's feedback mechanism. This is entirely the point of having markets provide this service. They're more responsive to the desires of consumers.

It's usually imperceptible in the real world if you're not actively looking for it, but markets very frequently settle on standard norms within an industry that all (or most) firms follow. Think about something as silly as the type of bag your potato chips come in. Nearly every firm in the potato chip industry uses a cellophane (I think that's what it is?) bag with basically the same physical properties. Why? Because the market structure tends to reflect the preferences of consumers. For whatever reason, consumers in the potato chip market prefer homogeneity of packaging, likely for reasons of convenience. There seems to be one "best" way of packing chips and so the market has settled on that method, for the most part (yes I'm aware of pringles).

In the case of there being a general consensus amongst the populace about some moral norm, competing courts would tend to also have consensus on that issue. In the case where there is a lack of general consensus amongst the populace about some moral norm, competing courts would also tend to have a lack of consensus on that issue, but this is also part of the beauty of the market solution. The lack of consensus of the courts reflects the lack of consensus of the people which reflects the fact that we, as a society, haven't figured this one out yet. The competing courts are the society's method of figuring it out.

I want to reply to your second point in more depth, but this has gotten ungodly long so I'll just briefly say that there are no assurances about the future; the future is by definition uncertain. I believe, as do many others, that there are compelling reasons why an ancap society could provide positive outcomes. The fact that we're not absolutely certain it would work out the way we hope is not a strong enough argument for definitively outlawing any attempt at experimenting with it. I'm not saying there should be a worldwide ancap revolution, just maybe allow a small "pilot program", a "special economic zone" for the ancap experiment.