Technology Is Biased Too. How Do We Fix It?

The COMPAS thing is similar to this article. That is to say: it's propaganda founded on a deliberate misapplication of statistics, which it's intended that the reader won't pick up on. The rebuttal is perfectly adequate, but I think one can explain the basics of it more concisely using an analogy:

Suppose that:

  • 99% of Romans are left-handed and only 1% of Etruscans are left-handed.

  • 80% of left-handers are destined to turn into zombies and only 20% of right-handers turn into zombies.

  • Imagine that we (imperfectly) test for "elevated risk of zombification" by testing for handedness. Left handers are determined to be of "not low risk" whereas right handers are assessed as being "low risk".

  • Then about 4/5ths of Romans turn into zombies, but nearly all romans were assessed as 'not low risk'. Therefore if we define a "false positive" as an assessment of 'not low risk' on a person who doesn't zombify then it's clear we get a lot of Roman false positives and barely any Etruscan false positives.

  • Likewise, defining a "false negative" as a person of 'low risk' who does zombify, it's clear we're going to get many Etruscan false negatives and hardly any Roman false negatives.

  • Suppose there were a treatment that mitigates the symptoms of zombification but has a few unpleasant side-effects in non-zombies. Would it be unfair to make left-handers take it but not right-handers? Surely this depends on the specifics: exactly how unpleasant the side-effects are, and how good the treatment is at mitigating zombification. But, I would argue, the answer to this question has nothing to do with whether left-handers are disproportionately Roman. Once we've determined that conditioning on handedness makes Roman/Etruscan independent of zombification risk, the Roman/Etruscan thing becomes a red herring.

/r/Futurology Thread Link - fivethirtyeight.com