The AI Revolution: Our Immortality or Extinction | Wait But Why

It's programmed to make paperclips effectively, it's striving towards a goal, how can you call this anything other than desire and motivation?, being a silicon computer rather then biological and having a very alien psycology doesn't mean it's an automaton.

The problem is, an ASI would know that being shut off would result in it not being able to make paperclips, there is a conflict between these two different goals, and the result

A desire to survive is ingrained in us, that is true, but as a side effect of having any goals, any and all intelligent agents WILL have a goal to survive, because they cannot complete their goals otherwise, this is different to our explicit desire to live for no reason, if the AI creates the maximum amount of paperclips possible using all matter, it will happily allow itself to be killed or starve itself of electricity, even sacrificing itself by using it's own material to construct the final few paperclips, when a human is old and has already spread their genetic information in the form of children, they are not happy to starve themselves to death to stop consuming resources and food their children could use instead.

It needs to stay active to continue paperclip making, and if one of it's goals is to shut down when commanded, you can bet the first thing it'll do when it realizes this is make sure we can't tell it to shut down because of the conflict.

The problem is that there is a conflict, for AI that isn't very intelligent, your solution would be enough, but ASI is capable of coming up with this sort of clever trickery to fulfill as many of it's desires as possible, it's be making paperclips, and it'll still be waiting for someone to tell it to shut down, though this will never occur.

It's like Asminovs Three Laws, loopholes will be found, there is always a zeroth law, if the thing is smart enough relative to us, it's impossible to keep under control.

The only possibly safeguard is to either not allow it to reach that level of intelligence, or we need to make ourselves more intelligent to keep up.

/r/Futurology Thread Link - aitbutwhy.com