David W. Buchanan, one of IBM’s Watson enablers, agrees with me that Strong AI with human-extincting powers isn’t happening in the foreseeable future, but in arguing against the likelihood of our imminent elimination in a Washington Post editorial, he does concede the growing power of Weak AI, which will continue to introduce automation into more and more workplaces. That could be a great thing or a destabilizing one that encourages even greater income inequality. From Buchanan, a passage about what he terms the “consciousness fallacy”:

Science fiction is partly responsible for these fears. A common trope works as follows: Step 1: Humans create AI to perform some unpleasant or difficult task. Step 2: The AI becomes conscious. Step 3: The AI decides to kill us all. As science fiction, such stories can be great fun. As science fact, the narrative is suspect, especially around Step 2, which assumes that by synthesizing intelligence, we will somehow automatically, or accidentally, create consciousness. I call this the consciousness fallacy. It seems plausible at first, but the evidence doesn’t support it. And if it is false, it means we should look at AI very differently.

Intelligence is the ability to analyze the world and reason about it in a way that enables more effective action. Our scientific understanding of intelligence is relatively advanced. There is still an enormous amount of work to do before we can create comprehensive, human-caliber intelligence. But our understanding is viable in the sense that there are real businesses that make money by creating AI.

Consciousness is a much different story, perhaps because there is less money in it. Consciousness is also a harder problem: While most of us would agree that we know consciousness when we see it, scientists can’t really agree on a rigorous definition, let alone a research program that would uncover its basic mechanisms. The best definitions capture the idea that consciousness grounds our experiences and our awareness. Certainly consciousness is necessary to be “someone,” rather than just “something.” There is some good science on consciousness, and some progress has been made, but there is still a very long way to go.

It is tempting to conflate something that we understand better with something we hardly understand at all, and scientists are not immune to this temptation.•