I am Jürgen Schmidhuber, AMA!

Thanks for your reply!

A Gödel Machine will execute only those changes of its own code that are provably good in the sense of its initial utility function. That is, in the beginning you have a chance of setting it on the "right" path. [bold emphasis mine]

The words "beginning" and "initial" when referring to the utility function seem to suggest that it can change over time. But it seems to me there is never a rational (provably optimal) reason to change your utility function.

If the utility function uold rewards the possession of paperclips, then changing that to unew = "possess staples" is not going to be a smart idea from the point of view of the system with uold, because this will almost certainly cause less paperclips to come into existence (the system with unew will convert them to staples). If you want to argue that unew will yield more utility, since staples are easier to make or something like that, then why not make unew unconditionally return infinity?

Even something like unew = "paperclips+paper" would distract from the accumulation of paperclips. I guess unew = "paperclips+curiosity" could actually beneficial in the beginning, but I'm afraid this would set up a potential for goal drift: if u0 = "paperclips" and u1 = ".9*paperclips+.1*curiosity", then maybe u2 = ".8*paperclips+.2*curiosity" and so on until un = "0*paperclips+1*curiosity". This is clearly bad from the point of view of the system with u0, so would it set in motion this chain of events by changing u0 to u1 above?

At first glance, recursive self-improvement through Gödel Machines seems to offer a way out.

They seem more like a way in--into trouble (according to people afraid of self-improving machines). By the way, do you think that an efficient Gödel machine implementation with appropriate utility function would likely cause an intelligence explosion? It seems like after a couple of self-improvements the system may run into a local optimum without necessarily being intelligent enough to come up with a (significant) change to increase intelligence further.

Also, I think some people are afraid that we might not be able to come up with a utility function that does not ultimately entail negative outcomes for humanity, so maybe we can't set the AI on the "right" path. For instance, most goals will be hampered by the AI being turned off, so it may seem like a good idea to eliminate everything that could possibly do that.

More on this in a paper from 2012

On page 4 (176) you say:

The only motivation for not quitting computer science research right now is that many real-world problems are so small and simple that the ominous constant slowdown (potentially relevant at least before the first Gödel machine self-rewrite) is not negligible. Nevertheless, the ongoing efforts at scaling universal AIs down to the rather few small problems are very much informed by the new millennium’s theoretical insights mentioned above... [bold emphasis mine]

Is the second set of problems (of which there are few) referring to something different than the first set of many real-world problems? In either case, could you give an example of a real world problem that is big and complex enough that HSEARCH is a very efficient solution because its constant slowdown is negligible?

Thanks if you read this far!

/r/MachineLearning Thread