Noise

Their central distinction, and the reason the book exists. Bias is systematic deviation in one direction; noise is unwanted variability in judgements that should be identical.

Their target metaphor makes it concrete: a team whose shots cluster off- centre is biased; a team whose shots scatter widely is noisy. Both produce error, and organisations spend almost all their attention on the first.

Bias is everyone missing in the same direction. Noise is everyone missing in different ones. Only one gets attention.

Devmark’s reading — not a quotation from the book

Their explanation for the neglect is structural. Bias can often be detected from outcomes; noise requires comparing judgements that were never made side by side.

Two underwriters price the same risk very differently, two judges sentence the same case very differently — but each sees only their own decision, so the variability leaves no trace anywhere in the organisation.

Their central practical instrument: give the same set of cases to multiple professionals independently and measure the spread.

Their reported results are startling — executives asked to guess the variability in their own organisations typically estimate around ten per cent, and audits routinely find several times that. The gap between expectation and measurement is the book's most persuasive evidence.

Level noise is variation in the average severity of different judges — some are simply harsher. Pattern noise is the stable idiosyncratic response of a particular judge to a particular kind of case. Occasion noise is variation within the same judge at different moments.

Pattern noise, in their data, is usually the largest component, which is uncomfortable because it is the part that feels most like professional judgement.

Their catalogue of what moves a judgement within the same person: mood, weather, time of day, hunger, the previous case, the order of presentation.

Each effect is small and none is defensible. Their point is not that professionals are frivolous but that a system which permits this much variability is producing outcomes that are partly arbitrary.

Their prescribed remedies, deliberately named by analogy with handwashing — you do not know which specific infection you prevented, but the practice works.

The components: aggregate independent judgements before discussion, structure complex decisions into separately assessed components, use a shared relative scale rather than absolute impressions, favour rules and algorithms where they are available, and sequence information so that early impressions cannot contaminate later assessments.

You never learn which error decision hygiene prevented. That is exactly why it has to be a standing practice.

Devmark’s reading — not a quotation from the book

Their review of the long literature comparing simple statistical models to expert judgement, which consistently finds the models perform at least as well and usually better.

Their explanation is largely noise: a model applied to identical inputs gives identical outputs, while an expert does not. The model is not smarter; it is consistent, and consistency alone accounts for much of the advantage.

They engage with the reasonable resistance: that reducing noise can entrench bias if the rule itself is biased, that algorithmic decision-making raises accountability and dignity concerns, and that some domains genuinely require judgement.

Their answer — that noise reduction and bias correction are separate problems that both need addressing — is sound, though the book is stronger on measurement than on the political questions.

It is long and repetitive, restating the core distinction many times across several hundred pages. The essential argument could be delivered in a third of the length, and most readers notice this.

The recommendations are also better suited to large organisations making repeated similar judgements than to individuals or to genuinely novel decisions, which the book acknowledges only briefly.

The noise audit is the transferable instrument, and it is cheap: give several colleagues the same case independently and compare. Almost every organisation that does this is surprised.

Then the sequencing discipline — collect independent judgements before anyone discusses, because the first opinion spoken aloud contaminates every one after it.