Yudkowsky compares the likelihood of catastrophe to a simple physical law.
If artificial intelligence is allowed to develop at its current pace, it will destroy humanity, believes AI researcher and rational thinking advocate Eliezer Yudkowsky. The problem is not only that a superintelligence could fall into the hands of 'bad actors.' According to Yudkowsky, regardless of who creates 'strong AI' first, we are all doomed. How this could happen is detailed in an excerpt from his book 'If Someone Creates It — Everyone Will Perish. Why Superhuman AI Will Destroy Us All.'
Yudkowsky's book is dedicated to 'All the people who have died throughout the long history of our species, to all who are still alive, and to all the children who may ever come into being.'
American philosopher, science popularizer, and co-founder of the non-profit Machine Intelligence Research Institute (MIRI), Eliezer Shlomo Yudkowsky did not graduate from either high school or university. He was born into an Orthodox Jewish family and became an example of a self-taught individual — someone who received an education entirely on their own. Today, he is one of the most prominent figures in AI research. Paradoxically, Yudkowsky has been calling for restrictions on AI development in recent years: in his view, uncontrolled technological progress poses existential threats.
Yudkowsky founded the Machine Intelligence Research Institute. Its current president, Nate Soares, co-authored their joint book with the apocalyptic title 'If Someone Creates It — Everyone Will Perish.'
The main thesis of the work, which resembles a political declaration, is that the race for superhuman AI will lead to the destruction of humanity. The key problem, the authors assert, is that no one — including the developers of modern generative AI — fully understands how it works.
Modern models are not 'created' but 'grown,' the book notes. This means that the potential consequences of technological development are beyond our control.
Yudkowsky compares the likelihood of catastrophe to a simple physical law: we know for certain that an ice cube thrown into water will melt, even if we cannot predict the trajectory of every molecule. Similarly, we do not know exactly how superintelligence will destroy humanity — whether it will cause the world's oceans to boil or trigger a new pandemic — but we know it will happen.
The co-authors of the book, along with hundreds of other scientists working in the field of artificial intelligence, signed an open letter consisting of one sentence: 'Reducing the risk of human extinction due to artificial intelligence must become a global priority — on par with pandemics, nuclear war, and other global-scale threats.' Now, Yudkowsky and Soares believe that they underestimated the scale of the problem at that time and propose a radical solution: to immediately halt all AI research worldwide.
An undeniable advantage of the book is its accessibility and simplicity of presentation. It will be understandable to both a fifth grader and your grandmother. The authors express their concerns as clearly as possible, and their arguments are presented sequentially.
At the same time, the persuasiveness of these arguments is questionable: in their polemical fervor, Yudkowsky and Soares cut several corners and commit logical fallacies. In any case, the issue of AI safety is undoubtedly more pressing now than ever, and the manifesto of technopessimists Yudkowsky and Soares confirms this by its very existence.
"Humanity faces an engineering challenge: how do we shape the preferences of artificial intelligence if we do not understand it? It does not matter whether a crowd of ethics specialists stands behind the engineers: ethics specialists also have no idea how to align the preferences of artificial intelligence with ours.
But discussing this engineering challenge is much less exciting than the problem of evil bosses ordering their AI models to make them god-emperors of Earth. Science fiction writers and Hollywood producers prefer tales of foolish corporate leaders rather than stories of AI models wanting strange things. Realism does not provide an exciting narrative.
If a screenwriter were tasked with writing a movie about machine superintelligence that begins to desire something strange, alien, and uncontrollable, they would immediately think about what dramatic and unexpected plot twists could embellish this idea. What if humans actually win? Perhaps the superintelligence will find a reason to keep us alive, free and healthy? Maybe, due to an unexpected turn of events, nothing bad will happen?
We believe that in reality, nothing resembling such a dramatic twist will occur. Such a film would be much sadder and significantly shorter."
