Book: Superintelligence - Paths, Dangers, Strategies

The back cover wrote:: If machine brains one day come to surpass human brains in general intelligence, the fate of our species would depend on the actions of powerful AI.

In this profoundly ambitious and original book, Nick Bostrom thoroughly examines what he calls the most essential problem of our time: can we solve the superintelligence control problem before it's too late?

Martin Rees, Past President of the Royal Society, wrote:: Those disposed to dismiss an 'AI takeover' as science fiction may think again after reading this original and well-argued book.

Nick Bostrom, in his preface, wrote:: If some day we build machine brains that surpass human brains in general intelligence, then this new superintelligence could become very powerful. And, as the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species would depend on the actions of the machine superintelligence.

We do have one advantage: we get to build the stuff. In principle, we could build a kind of superintelligence that would protect human values. We would certainly have a strong reason to do so. In practice, the control problem -- the problem of how to control what the superintelligence would do -- looks quite difficult. It also looks like we will get only one chance. Once unfriendly superintelligence exists, it would prevent us from replacing it or changing its preferences. Our fate would be sealed.

This is quite possibly the most important and most daunting challenge humanity has ever faced. And -- whether we succeed or fail -- it is probably the last challenge we will ever face.

I.J. Good, Alan Turing's chief statistician, in 1965, wrote:: Let an ultraintelligent machine be defined as a machine that car far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion," and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man ever need make, provided that the machine is docile enough to tell us how to keep it under control.

And it's all just far-fetched imagination.