What is malicious AI?

I think you've got the wrong idea about what an AI is to start with. An AI isn't just an if-then machine that is hard coded to react in certain ways to certain situations. An AI is a program that is able to "learn" and then react to new situations based on the training it has received.

The problem with AIs becoming malicious is that, even though we can train them to react to a wide range of situations, they are very psychologically different from people because we have millions upon millions of years of biological programming that makes us think and act in certain ways. We have a few primary goals built into us (pleasure, reproduction, survival, socialization, etc) and we form sub-goals based on those goals that still, in most cases, will follow within the guidelines of our primary goals.

However, an AI doesn't have those goals built into them like we do. In the famous paperclip maximizer example, we have an AI that is very, very smart, able to solve complex problems, have abstract thoughts, and so on. However, because it doesn't have the same biological programming built into it that we do, it makes decisions we would think are horrible, but that it thinks are great based on the primary goal we gave it - making more paperclips. It might decide to kill every human on Earth to stop us from interfering with paperclip production, or decide to turn the whole Earth into a giant ball of paperclips, for example.

For us, the goal of "make as many paperclips as possible" is still prioritized below our biological programming. We don't want to kill people to make more paperclips, we don't want to let ourselves die to make more paperclips, and so on. The machine doesn't care, it wants nothing more than to make as many paperclips as possible. That's what makes an AI like this "malicious".

/r/askscience Thread