The AI Superintelligence Debate Has Serious Flaws
Reported by Hacker News1 source ↓
The article questions whether recursive AI self-improvement can produce superintelligence, arguing that progress is constrained by the availability of suitable training data. It also warns that pursuing the goal could cause harm even if an intelligence explosion does not occur.
Key points
- The article argues that limited training data could constrain AI improvements outside fields such as math and programming
- Anthropic says its internal data shows AI is accelerating AI development
- OpenAI set a goal of developing an automated researcher by March 2028
Details
The article distinguishes artificial superintelligence from artificial general intelligence and questions whether engineers can build systems that improve themselves in an accelerating cycle. It says the idea dates back at least to statistician I.J. Good’s 1965 discussion of an “intelligence explosion” and gained traction among futurists in the 1990s.
The author points to a shift from scaling model size toward post-training methods, including reinforcement learning, which use examples of successful outputs to improve performance on particular tasks. Such examples are easier to produce in structured fields like programming and mathematics; the article argues that areas without comparable data may see slower progress.
Anthropic said in a June blog post that its internal data showed AI was already accelerating AI development, including through coding and research tasks. OpenAI stated in a September report that it aims to develop an automated researcher by March 2028. The article argues that coding agents and AI research tools would not remove the underlying data constraint.
Written from coverage as of
Images
- Hacker News
- Hacker News
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh