Human Compatible

Russell's central claim is that AI has been built on a model inherited from economics and control theory: specify an objective, then build a machine that maximises it.

His argument is that this is safe only when the objective is completely and correctly specified, and that we have never managed to completely and correctly specify what we want — which makes the model a mistake at the foundation rather than a problem at the margin.

The model is fine if the objective is perfectly specified. We have never once managed that.

Devmark’s reading — not a quotation from the book

Russell co-wrote the standard textbook that taught the standard model to a generation of students, which makes this an unusually credible act of self-criticism.

He addresses this directly: the textbook presents the framework he is now arguing against, and later editions have been revised to reflect the change. The candour is part of why the book is taken seriously inside the field.

His framing of misspecified objectives: getting exactly what you asked for and discovering it was not what you wanted.

His point is that this is not an exotic failure mode. It is the ordinary behaviour of optimisation, visible already in recommendation systems that maximise engagement and thereby optimise for outrage, addiction and polarisation without anyone having specified those.

His constructive proposal. The machine's only objective is to maximise the realisation of human preferences. The machine is initially uncertain about what those preferences are. The ultimate source of information about human preferences is human behaviour.

The second principle does the work. A machine uncertain about the objective has a positive incentive to ask, to defer, and to allow itself to be switched off — because being corrected improves its performance on the true objective.

His most elegant technical result, presented informally. A machine certain of its objective resists being switched off, because being off prevents the objective being achieved.

A machine uncertain of its objective treats a human reaching for the switch as evidence that it has misunderstood, and therefore permits it. Safety becomes a consequence of the design rather than a constraint bolted on afterwards.

A machine certain of its goal resists shutdown. An uncertain one treats your hand on the switch as information.

Devmark’s reading — not a quotation from the book

The technical machinery behind the proposal: rather than being given a reward function, the machine infers preferences from observed behaviour.

Russell is honest about the difficulties — humans are inconsistent, act against their own stated preferences, are myopic, and differ from each other — and treats these as research problems rather than as objections that dissolve the approach.

His chapter on the difficulties is the most intellectually serious part. Whose preferences? How are conflicting preferences aggregated? What about preferences that are themselves shaped by the system? What about people who want things that harm others?

He does not resolve these. Presenting them clearly as open problems rather than as solved is what distinguishes the book from advocacy.

His material on present harms — surveillance, manipulation via recommendation, autonomous weapons, the labour effects of automation — is concrete and connects the long-horizon argument to documented current issues.

This is the part that answers the common objection that existential-risk work distracts from present problems, since Russell treats them as the same misspecification problem at different scales.

Critics argue that the standard model is a caricature of what practitioners actually do, that modern systems are not well described as objective maximisers, and that the proposal is philosophically appealing and technically underspecified.

The preference-aggregation problem in particular is a very old problem in social choice theory with known impossibility results, and the book does not fully engage with how severe that is.

The reframing is the contribution: that safety is a property of how the objective is treated rather than a set of restrictions applied to a finished system.

And the uncertainty principle, which is useful well beyond machines — a system confident it knows what you want behaves very differently from one that treats your correction as information.