Why Computers Won't Make Themselves Smarter

agiintelligence-explosionsingularityrecursive-self-improvementai-criticism

Abstraction: Skeptical argument against recursive AI self-improvement and intelligence explosion

Key points:

Ted Chiang argues I.J. Good's 1965 "ultraintelligent machine" definition does the same logical work as Anselm's ontological argument — it defines capability into existence rather than demonstrating it
Compiler bootstrapping is the only concrete example of a self-improving program, and it hits a hard ceiling: CompilerTwo cannot produce CompilerThree; all further optimization required human programmers
The analogy of running a human-equivalent AI 100x faster for a year (~100 person-years) would yield output like writing Windows XP, not a smarter AI — major breakthroughs require research insight, not just time
C. elegans (302 neurons, fully mapped) is still not fully understood behaviorally, suggesting self-understanding doesn't scale — the brain needed to understand a system may need to be larger than the system
Recursive improvement does happen at the level of human civilization through accumulated cognitive tools (Arabic numerals, calculus, computers), not individual intelligence
Replicating the innovation value of all humanity would require ~8 billion human-equivalent AIs, not one superintelligence, making "intelligence explosion" economically implausible

Connections: Ted Chiang · Artificial General Intelligence · Intelligence Explosion · AI Safety