AI and Greediness

The question is what to do about the arms race dynamic.

The most obvious and most proposed solution is to create a shared-source AI project and bring the world's known major players on board. Besides the basic political impossibility of accomplishing that (getting every company to fully give up important trade secrets, getting every international rival to fully cooperate, etc), there is also the danger that it could lead to the unintended proliferation of dangerous technologies. Every person that is given access to a sensitive software development project is another potential leak, and there is no putting the genie back in the bottle. Given that general intelligence capability in particular is likely to be useful in a weapons capacity, the idea of making it semi-open source is potentially very scary, even under the most ideal circumstances. It will have to be a relatively secret project, and the people on board will have to be high quality people, because the nature of the product is highly potent and potentially highly proliferable. You can potentially duplicate a WMD with copy/paste. We are already running into that with cyberwarfare before we even get to AI.

We knew that nuclear weapons could lead to the destruction of civilization, but we didn't open source them to address that concern (bit of a false comparison I admit, but still an important lesson in the political dynamics of powerful technologies). Instead, the initial developer tried to corner the market. We're almost certain to see a repeat of that no matter what policies are in place to stop it. We might do better to try to minimize negatives in that scenario rather than expending political capital on pie-in-the-sky collaborative efforts.

That would look something like encouraging the efforts of the "good guys," whoever you think that might be, and crossing your fingers. The arms race dynamic might actually be the dynamic we want, as long as the runners don't run dangerously fast.

/r/ControlProblem Thread