Weapons of Math Destruction

O'Neil's definition of the systems she is attacking is precise, which distinguishes the book from general algorithm anxiety. A weapon of math destruction is opaque, scalable, and damaging.

Opaque: those affected cannot see how they were scored. Scalable: it touches very large numbers of people. Damaging: being scored badly makes the outcome worse rather than merely predicting it. A model failing any of the three is not her target.

Opaque, scalable, damaging. A model that fails any one of the three is not what the book is attacking.

Devmark’s reading — not a quotation from the book

Her central technical point: these systems cannot measure what they care about, so they measure something correlated — postcode for creditworthiness, school attended for capability, keystroke patterns for reliability.

Her argument is that proxies frequently encode existing inequality, so a model using them reproduces the inequality while appearing neutral, and the appearance of neutrality is what makes it harder to challenge.

Her most important structural argument. A model that predicts crime in an area directs policing there, which produces more recorded crime there, which confirms the model.

The loop means the model creates its own evidence. It is right because it was acted on, not because it was accurate, and there is no signal inside the system that would reveal the difference.

Her related point about why these systems do not improve: a rejected loan applicant, a teacher fired by an algorithm, or a candidate filtered out of a hiring pipeline never generates the outcome data that would show the model was wrong.

Successful predictions are observed and errors are invisible, which means the model's apparent accuracy is measured on a censored sample.

Her domains: university rankings and the behaviour they induced, predatory advertising aimed at people in financial distress, personality tests in hiring, scheduling software that destabilised workers' lives, recidivism scoring in sentencing, credit scoring, and insurance pricing.

The scheduling chapter is the most under-discussed and among the most damning, because the harm is entirely a side effect of optimising a cost the worker does not appear in.

The scheduling chapter is the quietest and the worst: the harm is a side effect of optimising a cost workers do not appear in.

Devmark’s reading — not a quotation from the book

Her argument that a flawed human decision affects one person while a flawed model affects millions, applying the same error consistently.

Human bias is distributed and inconsistent; model bias is uniform and systematic, which is worse for the person on the wrong side of it and better for anyone wanting to detect it — a tension she notes without fully resolving.

Her recommendations: auditing, transparency requirements, regulation analogous to other consequential industries, and a professional ethic for data scientists comparable to those in medicine or engineering.

The proposals are less developed than the diagnosis, which she partly acknowledges. Her Hippocratic-oath suggestion in particular has been criticised as insufficient against structural incentives.

The book is polemical and the case studies are selected to make the argument. There is little discussion of algorithmic systems that reduced discrimination relative to the human process they replaced, which does happen.

Technically it is also light — O'Neil is a mathematician writing accessibly, and readers wanting the fairness literature will need to go elsewhere, including to Christian's book.

Extremely well in the wrong way: published in 2016, it described problems that have since become considerably larger with the deployment of much more capable systems.

Several of its specific cases have since been the subject of regulation, litigation or public reversal, which is a reasonable indication that the diagnosis was correct.

The three criteria, used as a checklist on any scoring system: can the person see how they were scored, how many people does it touch, and does a bad score make their situation worse?

And the feedback-loop question — does acting on this model generate the data that will confirm it? — which is the single most useful thing to ask about any deployed prediction.