Technically Exists

Superintelligence reference page

Introductions

Superintelligence is a very broad and complex topic. These resources are for those who are mostly or entirely unfamiliar with the subject.

• The Superintelligence FAQ by Scott Alexander of Slate Star Codex is an introduction to the dangers of superintelligence in the form of answers to “frequently asked questions”.
• Tim Urban of Wait But Why writes about superintelligence in two parts.
  • The AI Revolution: The Road to Superintelligence deals with how we might get to superintelligence from the AI systems we have now.
  • The AI Revolution: Our Immortality or Extinction discusses the possible consequences of superintelligence.
  • There’s also A reply to Wait But Why on machine superintelligence by Luke Muehlhauser, which clarifies and corrects various details of the two posts.
• Smarter Than Us is a book by Stuart Armstrong explaining why AI is likely to eventually become smarter than humans and the problems that this poses.
• Extinction Risk from Artificial Intelligence is a website created by Michael Cohen as an introduction to superintelligence and the risks it poses to the human species.

Alignment problem

The alignment problem is the problem of aligning an AI’s goals with the values of humanity. Solving this problem is often believed to be critical in ensuring that superintelligence has a positive impact on the world.

• AI Alignment: Why It’s Hard, and Where to Start is a talk by Eliezer Yudkowsky available in both video and transcript form. It covers topics like subproblems in AI alignment, why alignment is both difficult and necessary, and what progress has already been made.
• Yudkowsky also has a shorter post that covers the necessity of alignment, the difficulty of alignment, and why the alignment problem is not self-solving.
• Of Myths And Moonshine by Stuart Russell articulates the case for emphasizing the alignment problem in AI research.

Intelligence explosion

An intelligence explosion is a hypothetical scenario in which a self-improving intelligence is able to improve itself more and more rapidly thanks to the very improvements it makes. The feasibility of this scenario has great implications for how superintelligence may arise.

• MIRI’s Intelligence Explosion FAQ answers questions regarding the intelligence explosion concept, superintelligence, and other related ideas.
• Intelligence Explosion: Evidence and Import by Luke Muehlhauser and Anna Salamon argues that it is feasible for human-level AI to be developed before 2100 and for an intelligence explosion to follow shortly after.

Slow takeoff

A slow takeoff, sometimes called a soft takeoff, is a scenario in which a self-improving intelligence improves itself slowly enough for humans to intervene during the process. This is the opposite of the fast or hard takeoff experienced in the intelligence explosion scenario.

• Victoria Krakovna’s Risks from general artificial intelligence without an intelligence explosion lists reasons why an AI improvement scenario lacking an intelligence explosion still comes with many risks.
• Decisive Strategic Advantage without a Hard Takeoff by Kaj Sotala explores the possibility of an AI acquiring control of the world absent an intelligence explosion.
• Existential risk from AI without an intelligence explosion by Alex Mennen covers reasons an AI might gain a decisive strategic advantage prior to undergoing an intelligence explosion and how such scenarios differ from the reverse situation.

Orthogonality thesis

The orthogonality thesis states that, barring a few edge cases, any level of intelligence can be combined with any terminal goal. This precludes scenarios where, for example, sufficiently smart superintelligences will automatically replace any arbitrary goals they were initially given with a goal of behaving morally.

• Bostrom lays out the orthogonality thesis in his paper The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents.
• Arbital has an Orthogonality Thesis page that covers arguments for and implications of the orthogonality thesis.
• The Orthogonality Thesis, Intelligence, and Stupidity is a video by Robert Miles that explains the orthogonality thesis using Hume’s guillotine.

Instrumental convergence thesis

The instrumental convergence thesis states that there are instrumental goals that will be useful to agents with a wide variety of terminal goals. Self-preservation, resource acquisition, and self-improvement are all examples of convergent instrumental goals.

• The Basic AI Drives by Stephen Omohundro lays out the reasoning behind why a wide range of terminal goals would lead to similar instrumental goals, establishing the groundwork for the instrumental convergence thesis.
• Bostrom’s The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents formulates the instrumental convergence thesis alongside the orthogonality thesis.
• Arbital’s Instrumental convergence page gives a fairly technical introduction to the claims made by the instrumental convergence thesis.
• Miles explains why convergent instrumental goals arise and how this allows powerful agents’ behaviors to be predicted in the video Why Would AI Want to do Bad Things? Instrumental Convergence.

Timelines

When will superintelligence be created? The answer to this question is important because it determines how long there is to prepare for its arrival. As such, it is worth looking to predictions of when various AI milestones will be reached.

• AI Impacts has a Guide to pages on AI timeline predictions which is primarily focused on timelines for pre-superintelligence milestones but also includes some superintelligence timelines.
• Future Progress in Artificial Intelligence: A Survey of Expert Opinion by Vincent C. Müller and Nick Bostrom includes questions about timelines for “high–level machine intelligence” and superintelligence.