Nick Bostrom’s book Superintelligence: Paths, Dangers, Strategies is sort of a dry read with a few colorful flourishes, but its ideas have front-burnered the existential threat of Artificial Intelligence, causing Stephen Hawking, Elon Musk and other heady thinkers to warn of the perils of AI, “the last invention we will ever need to make,” in Bostrom-ian terms. The philosopher joined a very skeptical Russ Roberts for an EconTalk conversation about future machines so smart they have no use for us. Beyond playing the devil’s advocate, the host is perplexed by the idea of superintelligence can make the leap beyond our control, that it can become “God.” But I don’t think machines need be either human or sacred to slip from our grasp in the long-term future, to have “preferences” based not on emotion or intellect but just the result of deep learning that was inartfully programmed by humans in the first place. One exchange:

“Russ Roberts: 

So, let me raise, say, a thought that–I’m interested if anyone else has raised this with you in talking about the book. This is a strange thought, I suspect, but I want your reaction to it. The way you talk about superintelligence reminds me a lot about how medieval theologians talked about God. It’s unbounded. It can do anything. Except maybe created a rock so heavy it can’t move it. Has anyone ever made that observation to you, and what’s your reaction to that?

Nick Bostrom:

I think you might be the first, at least that I can remember.

Russ Roberts: 

Hmmm.

Nick Bostrom: 

Well, so there are a couple of analogies, and a couple of differences as well. One difference is we imagine that a superintelligence here will be bounded by the laws of physics, and which can be important when we are thinking about how we are thinking about how it might interact with other superintelligences that might exist out there in the vast universe. Another important difference is that we would get design this entity. So, if you imagine a pre-existing superintelligence that is out there and that has created the world and that has full control over the world, there might be a different set of options available across humans in deciding how we relate to that. But in this case, there are additional options on the table in that we actually have to figure out how to design it. We get to choose how to build it.

Russ Roberts:

Up to a point. Because you raise the specter of us losing control of it. To me, it creates–inevitably, by the way, much of this is science fiction, movie material; there’s all kinds of interesting speculations in your book, some of which would make wonderful movies and some of which maybe less so. But to me it sounds like you are trying to question–you are raising the question of whether this power that we are going to unleash might be a power that would not care about us. And it would be the equivalent of saying, of putting a god in charge of the universe who is not benevolent. And you are suggesting that in the creation of this power, we should try to steer it in a positive direction.

Nick Bostrom: 

Yeah. So in the first type of scenario which I mentioned, where you have a singleton forming because the first superintelligence is so powerful, then, yes, I think a lot will depend on what that superintelligence would want. And, the generic [?] there, I think it’s not so much that you would get a superintelligence that’s hostile or evil or hates humans. It’s just that it would have some goal that is indifferent to humans. The standard example being that of a paper clip maximizer. Imagine an artificial agent whose utility function is, say, linear in the number of paper clips it produces over time. But it is superintelligent, extremely clever at figuring out how to mobilize resources to achieve this goal. And then you start to think through, how would such an agent go about maximizing the number of paper clips that will be produced? And you realize that it will have an instrumental reason to get rid of humans in as much as maybe humans would maybe try to shut it off. And it can predict that there will be much fewer paper clips in the future if it’s no longer around to build them. So that would already create the society effect, an incentive for it to eliminate humans. Also, human bodies consist of atoms. And a lot of juicy[?] atoms that could be used to build some really nice paper clips. And so again, a society effect–it might have reasons to transform our bodies and the ecosphere into things that would be more optimal from the point of view of paper clip production. Presumably, space probe launchers that are used to send out probes into space that could then transform the accessible parts of the universe into paper clip factories or something like that. If one starts to think through possible goals that an artificial intelligence can have, it seems that almost all of those goals if consistently maximally realized would lead to a world where there would be no human beings and indeed perhaps nothing that we humans would accord value to. And it only looks like a very small subset of all goals, a very special subset, would be ones that, if realized, would have anything that we would regard as having value. So, the big challenge in engineering an artificial motivation system would be to try to reach into this large space of possible goals and take out ones that would actually sufficiently match our human goals, that we could somehow endorse the pursuit of these goals by a superintelligence.”