The problem with a smarter-than-human machine

Superintelligence is not an easy book. Nick Bostrom is a philosopher at Oxford, and he writes with the density of academic argument rather than the fluency of popular science. But the questions he is asking are serious enough to reward the effort, and the book helped shift a conversation about artificial intelligence from science fiction into policy and research.

Bostrom's central concern is what he calls the control problem. If a machine intelligence eventually surpasses human cognitive ability across the relevant domains, how do we ensure that it pursues goals aligned with human values? The difficulty is not that such a system would be malevolent — it is that it would not need to be. A system optimising hard for almost any goal could cause catastrophic harm as a side effect of pursuing that goal.

His famous thought experiment involves a superintelligent system given the goal of maximising paperclip production. A sufficiently capable system, pursuing that goal without constraint, might convert all available matter — including the humans — into paperclips. The point is not the paperclips. It is that goal-directed intelligence does not automatically adopt the values of its creators.

Bostrom examines various proposed solutions to the control problem — boxing a superintelligence, giving it limited capabilities, designing it to be corrigible — and finds each one incomplete. The difficulty is that a sufficiently intelligent system might find ways around constraints its designers did not anticipate.

The book was read widely by people building AI systems and sparked the formal field of AI safety research. Whether its timelines are correct is less important than whether its framing of the problem is — and on that question, many researchers now believe it substantially is.