Personally, I follow Solomonoff's approach because I do not think that emulating human or animal behavior is the fundamental problem of AI.
Okay so if we look at the very bottom section of the human large intestine, there is a valved sack called the appendix. The function of this part of our body is not understood by any scientific consensus. But look at the popular evolutionary explanation: That the human appendix is a vestigial organ once used by ancient primate ancestors who probably digested leaves.
Now that type of thinking is not considered wildly unreasonable by people of today. Now I would ask Shane Legg if the same sort of evolutionary reasoning should extend "above the neck" as it were?
We want to kind of avoid the idea that homo sapiens evolved, but only "from the neck down". And that above the neck, there is some pure, unadulterated blank slate which is a very powerful, bayesian inference engine, inexplicable in its natural origins.
Shane Legg provides the aphorism:
Can we counter with the human brain is in no sense general? Is that some sort of intellectual foul in 2015? If we are going to extend the word "generality" to "the set of all recursive functions on a Turing machine, weighted by priors given by Kolmogorov complexity" we have extended our net to cosmic proportion. In that context, the human brain is a narrow collection of evolved heuristic hacks, promoted by reproduction in some of socio-linguistic context of African savannah. .. only when extending out context that far.
If we consider the length and encoding scheme of entire genotypes (the largest of which is found in a plant in Japan) , the relationship between the process of natural selection and Kolmogorov Complexity is not understood by science. There is a very good reason it is not understood. The rate of mutation of genes is dependent on copying errors from molecular biochemistry. The factors effecting that process are only vaguely understood in toy models. More clearly, what I am saying here is that we cannot take for granted that natural selection was "weighing" the probability of variations in a population on the basis of Kolmogorov Complexity.
Heuristic hacks are not useless. A discovery of what they are, how they function, and building technology that mimics that function could plausibly change human history.
Your thoughts...