Superforecasting

The book builds on Tetlock's long study of expert political judgement, which found that the average expert forecast was barely better than chance and that media prominence correlated negatively with accuracy.

That work produced the hedgehog-and-fox distinction: experts with one big organising idea performed worse than those who drew on many frameworks and held them loosely.

The more confident and media-prominent the expert, the worse the forecasts. That was the starting point.

Devmark’s reading — not a quotation from the book

The Good Judgment Project entered a government-sponsored forecasting competition with thousands of volunteers making probabilistic predictions about geopolitical questions, scored with a proper scoring rule.

The design is what makes the findings credible: questions resolved unambiguously, forecasts were numeric, and accuracy was measured rather than asserted. Most claims about forecasting ability have none of these.

The identified top performers were not domain experts and did not have access to classified information. What they had was a method.

They broke questions into components, started from base rates, updated frequently in small increments, sought disconfirming information deliberately, worked in teams that challenged each other, and expressed uncertainty in fine-grained numbers rather than in words.

One of the more surprising empirical findings: forecasters who used finer-grained probabilities were measurably more accurate, and rounding their forecasts to coarser buckets degraded performance.

This suggests the precision was carrying real information rather than being false confidence — a result that argues against the common instinct to express uncertainty vaguely.

Their treatment of belief revision is the most practically demanding part. Superforecasters updated more often than others and in smaller steps, avoiding both anchoring on an initial view and swinging on each new headline.

The skill is calibrating how much a piece of information should move you, which is genuinely difficult and is where most of the performance gap appears.

They updated more often and moved less each time. Both halves are necessary, and the second is harder.

Devmark’s reading — not a quotation from the book

Their most transferable instruction: begin with the base rate for the class of event before considering the specifics of this case.

The specifics feel more informative and usually are not. Starting outside and adjusting inward outperforms starting from the particulars, which is the same finding Kahneman reports from a different direction.

The book's underlying argument is institutional rather than personal: forecasting cannot improve without scoring, and almost no organisation scores its forecasts.

Tetlock is pointed that pundits face no accountability because predictions are made vaguely enough to be unfalsifiable — and that the fix is requiring numbers and dates, which immediately changes what people are willing to say.

They are honest about scope. The tournament questions had resolution dates within months to a couple of years; nobody demonstrated skill at long-range forecasting, and Tetlock does not claim it exists.

The findings also concern questions with definable outcomes. They say little about genuinely novel situations, and Taleb's objection — that the consequential events are the ones outside any base rate — is acknowledged rather than answered.

Like most books of this shape it states its findings early and then illustrates them at length. The profiles of individual forecasters add texture and not much argument.

The core content would fit comfortably in half the pages, which readers frequently notice around the midpoint.

Three practices transfer immediately. Put a number and a date on any prediction you make. Start from the base rate before the specifics. Update in small increments and record why.

And the institutional version: if your organisation makes forecasts and does not score them, it has no way of knowing whether it is any good, and almost certainly assumes it is.