The Paradox of a Smart World

We live surrounded by new ways of thinking, understanding, and measuring that simultaneously point to a new step in human evolution and an evolution beyond humans.

Some of the new ways of thinking involve data — mapping, matching, and searching for patterns far beyond the capacity of the human eye or ear. Some involve analysis — supercomputers able to model the weather, play chess, or diagnose diseases (for example, using the technologies of firms like Google's DeepMind or IBM's Watson). Some pull us ever further into what the novelist William Gibson described as the "consensual hallucination" of cyberspace.

These all show promise. But there is a striking imbalance between the smartness of the tools we have around us and the more limited smartness of the results. The Internet, World Wide Web, and Internet of things are major steps forward in the orchestration of information and knowledge.

Yet it doesn't often feel as if the world is all that clever. Technologies can dumb down as well as smarten up. Many institutions and systems act much more stupidly than the people within them, including many that have access to the most sophisticated technologies. Martin Luther King Jr. spoke of "guided missiles but misguided men," and institutions packed with individual intelligence can often display collective stupidity or the distorted worldview of "idiots savants" in machine form. New technologies bring with them new catastrophes partly because they so frequently outstrip our wisdom (no one has found a way to create code without also creating bugs, and as the French philosopher Paul Virilio put it, the aircraft inevitably produces the air disaster).

In the 1980s, the economist Robert Solow commented, "You can see the computer age everywhere but in the productivity statistics." Today we might say again that data and intelligence are everywhere — except in the productivity statistics, and in many of the things that matter most. The financial crash of the late 2000s was a particularly striking example. Financial institutions that had spent vast sums on information technologies failed to understand what was happening to them, or understood the data but not what lay behind the data, and so brought the world to the brink of economic disaster. In the 1960s and 1970s, the Soviet government had at its disposal brilliant minds and computers, but couldn't think its way out of stagnation. During the same period, the US military had more computing power at its disposition than any other organization in history, but failed to understand the true dynamics of the war it was fighting in Vietnam. A generation later the same happened in Iraq, when a war was fought based on a profound error of intelligence launched by the US and UK governments with more invested than any other countries in the most advanced intelligence tools imaginable. Many other examples confirm that having smart tools does not automatically lead to more intelligent results.

Health is perhaps the most striking example of the paradoxical combination of smart elements and often-stupid results. We now benefit from vastly more access to information on diseases, diagnoses, and treatments on the Internet. There are global databases of which treatments work; detailed guidance for doctors on symptoms, diagnoses, and prescriptions; and colossal funds devoted to pushing the frontiers of cancer, surgery, or pharmaceuticals.

But this is far from a golden age of healthy activity or intelligence about health. The information available through networks is frequently misleading (according to some research, more so than face-to-face advice). There are well over 150,000 health apps, yet only a tiny fraction can point to any evidence that they improve their users' health. The dominant media propagate half-truths and sometimes even lies as well as useful truths. And millions of people make choices every day that clearly threaten their own health. The world's health systems are in many ways pioneers of collective intelligence, as I will show later, but much doesn't work well. It's estimated that some 30 to 50 percent of antibiotic prescriptions are unnecessary, 25 percent of medicines in circulation are counterfeit, somewhere between 10 and 20 percent of diagnoses are incorrect, and each year 250,000 die in the United States alone because of medical error (the third leading cause of death there). In short, the world has made great strides in improving health and has accumulated an extraordinary amount of knowledge about it, yet still has a long way to go in orchestrating that knowledge to best effect.

Similar patterns can be found in many fields, from politics and business to personal life: unprecedented access to data, information, and opinions, but less obvious progress in using this information to guide better decisions. We benefit from a cornucopia of goods unimaginable to past generations, yet still too often spend money we haven't earned to buy things we don't need to impress people we don't like.

We have extraordinary intelligence in pockets, for specific, defined tasks. Yet there has been glacial, if any, progress in handling more complex, interconnected problems, and paradoxically the excitement surrounding new capacities to sense, process, or analyze may distract attention from the more fundamental challenges.

In later chapters, I address what true collective intelligence would look like in some of the most important fields. How could democracy be organized differently if it wanted to make the most of the ideas, expertise, and needs of citizens? Various experiments around the world suggest what the answers might be, but they baffle most of the professionals brought up in traditional politics. How could universities become better at creating, orchestrating,...