Why? If the alternative is no security at all, and there is no free baseline for the providing of such, all a monopoly needs to do is literally being better than nothing.

That issue already exists in the government monopoly. A monopoly is a monopoly, and it will tend to exist monopoly behaviour. Their monopoly status will make them tend to offer a poorer product at a higher cost relative to their behaviour in the presence of active competition. The separate issue is how are they financed.

In the case where of the private monopoly, your choice as a "consumer" is between paying a high price for a poor product, or refraining from consuming that product and instead consuming a different, possibly unrelated product (i.e. the next highest consumption bundle on your utility scale if we want to look at it from a theoretical economic point of view). That is to say, you can still choose not to give them your money. In the case of the government monopoly however, your choice is between paying a high price for a poor product and that's it. You are compelled by force of law to purchase the product. Additionally, they generally (though not always) outlaw even attempting to provide the product privately.

This provides an entirely new poor set of incentives for the institution which already suffers from poor incentives (the monopoly), making a bad institution tend to be even worse.

As for defending yourself, other than the problem of most people being untrained to handle something like a robbery, there is also the problem that they wouldn't be able to watch over their property and any defenseless loved ones all the time.

I'm not saying perpetual self-defense is the way the society should handle the problem of violent crime. I'm saying regardless of the type of society, there are two options: defend yourself or fail to defend yourself and pursue them through the criminal justice system. In the overwhelming majority of cases the latter is used rather than the former in either type of society. The police, for the most part, do not stop crimes in progress; they investigate and apprehend people who have broken laws after the fact. The same would likely be true in a free society. People do however defend themselves, in some societies frequently.

But supposing that it does work, how is the treatment of criminals decided?

Nobody can answer definitively how the treatment of criminals would be decided in an ancap society instead of statist society. The treatment of criminals is a social issue which is mostly reflected in the moral attitudes of the society. I can tell you what types of ideas are likely to be consistent with an ancap society.

The moral attitudes I could see being prevalent would see the role of criminal punishment to be based around the idea of restitution rather than being punitive. If someone breaks into your house and steals something for you he should be made to repay you its value. The definition of value is open to interpretation but I'm sure there are many reasonable standards the society could settle on that would tend to become dominant in most peoples' thinking.