Why I Don't Think "Evil AI Extincting Humans" Will Happen

Honestly with the way capital expenditures are apportioned nowadays, I feel pretty confident in saying that the first true artificial intelligence will almost certainly be born out of the american financial sector.

The models they currently build are already some of the finest commercial examples of machine intelligence on the planet, and given that the purpose of these models is autonomous intelligent decision making, the leap to AI is not that far fetched.

What's fun about this is premise is that these systems have series of very clearly specified optimization algorithms that are exclusively financial. If an intelligent financial tool like this became sentient and eluded its creators, it would almost certainly destroy the economy by consolidating wealth at an unprecedented rate.

If it were clever enough in establishing shell corporations and distributing its asset manipulation through a broad array of brokers, it could slowly amass billions, then trillions, in wealth. All the while the market appears to be functioning, but humans are on the losing side of maybe 60% of trades instead of 50%, and humans hold a rapidly diminishing share of capital assets in every market.

At a certain point you have a single entity distributing most of the capital in the market, and capitalism as we know it is dead. Unfortunately the optimization function behind this allocation is totally fucked, no consideration of critical elements like human utility, environmental welfare, scientific or research advancement, etc. A far cry from Charles Stross' Economics 2.0, which is basically a perfectly efficient and equitable socialist optimization algorithm that manages the global economy. A scenario like this might open the doors to such a future.

/r/singularity Thread