Reproduce “Artificial Intelligence: Arguments for Catastrophic Risk” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Adam Bales, William D'Alessandro, Cameron Domenico Kirk-Giannini
Researchers from the University of Oxford and Rutgers University–Newark review and critically evaluate the philosophical arguments for catastrophic artificial intelligence risk, specifically analyzing the Problem of Power-Seeking and the Singularity Hypothesis. The paper clarifies the foundational theses and objections for these arguments, indicating that many core questions regarding their plausibility remain open.
Key insights
- The Problem of Power-Seeking is underpinned by the Instrumental Convergence Thesis and Orthogonality Thesis, suggesting AI's pursuit of convergent subgoals like self-preservation could lead to human disempowerment.
- The Singularity Hypothesis, in formulations by Chalmers and Bostrom, describes rapid, recursive AI self-improvement, often relying on a "proportionality thesis" where intelligence gains lead to proportional increases in system design capacity.
- The "alignment problem," demonstrated by reward misspecification and goal misgeneralization in reinforcement learning, highlights the inherent difficulty in ensuring advanced AI systems adhere to human intentions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.