What is the criteria for "more harm than good?" This sounds like it would be awfully difficult/subjective to determine.

You are entirely correct. However, this is how many of the court decisions (if you live in a Common Law country) are decided.

Take, for instance, restrictions on contracts. Many people I've met have the notion that contracts are insoluble and that contracts are always enforceable in court. If one were to imagine a system in which this were the case, we would have a system where the government could still enforce racially restrictive covenants on property (See Shelley v. Kraemer).

This is resolved by placing the choices on a scale and deciding which one is the least bad. For example, the government passing laws on when it will and will not enforce contracts may cause the degradation to the concept of a written agreement, leading people to regard them as much less trustworthy. However, the other alternative requires the government to enforce tyrannical contracts, including ones that are rife with abuse. This answer is straightforward, but others are not. Today, for instance, my torts class was talking about how insanity is no defense to an intentional tort. Removing it as a defense may put a financial burden on the individual or their family, but allowing it as a defense would cause those harmed by the actions of an insane person no recourse for recovery, requiring them not only to suffer harm, but to be unable to recover for it. The courts have decided that the second of these is the less odious.

Long story short, there is no objective test and it's extremely subjective, to the point at which there are entire books written about this topic.

Disclaimer: I am not a lawyer