If Anyone Builds It, Everyone Dies

"Insofar as the AI has weird alien preferences, escape is in fact the course of action that best fulfills its objectives. Attempts to escape are not a weird personality quirk that an engineer could rip out if only they could see what was going on inside; they are generated by the same dispositions and capabilities that the AI uses to reason, to uncover truths about the world, to succeed in its pursuits."

"But if someone has read the history of engineering disasters, they should quickly realize this phase of the standard template for disaster. It’s the part where the most informed and most worried parties have to downplay their fears, because the rest of the system hasn’t caught up, and others would give them strange looks…. History is full of other examples of catastrophic risk being minimized and ignored…."

"Speed is often better, but AI is different from nearly every problem we’ve faced so far. When missteps kill everyone, you can’t just run fast and accept a few early mistakes."

"The experts in this field argue in opaque academic terms about whether everyone on earth will die quickly (our view); versus whether humanity will be digitized and kept as pets by AIs that care about us to some tiny but nonzero degree; versus whether there’s a 20 percent chance we die, and an 80 percent chance that superintelligence will be harnessed successfully by a corporation, which will then be able to wield its power as they see fit.… When these are the debates experts are having, you don’t have to be certain which experts are right to understand that the current situation is not okay."

"The issue is not that AIs will desire to dominate us; rather, it’s that we are made of atoms they could use for something else."

"Problems like this are why we say that if anyone builds it, everyone dies. If all the complications were visible early, and had easy solutions, then we’d be saying that if any fool builds it, everyone dies, and that would be a different situation. But when some of the problems stay out of sight? When some complications inevitably go unforeseen? When the AIs are grown rather than crafted, and no one understands what’s going on inside of them? That’s not a problem that anyone’s equipped to solve."