But AI is Different...

The crux behind the superintelligence debate

My friend wrote this take and wanted me to post it here. It’s a good take, give it a read!

---

AI is different. Not different in degree but different in kind: extreme enough that the precedent doesn’t carry over. The reasoning and counterarguments we apply to current humans, or to other intelligent beings, simply don’t apply, because AI is extremely different. Three of the most common cases show the shape of it.

The pattern

Each case: an empirical/historical argument that risk is overstated.

• Control. We’ve controlled coerced populations for millennia, so we can control AIs. → Different: slaves could revolt yet had broadly human goals and limited power; superintelligence is more capable and more alien.

• Optimization. Humans and chimps aren’t strict optimizers, and strict optimization may not even be coherent given sparse data and the need to generate hypotheses from priors — so why expect it of AI? → Different: enough intelligence finds proxies to optimize and workarounds we can’t foresee.

• Recursive self-improvement. Current systems show none — no firm fires its coders to buy compute; the binding constraints are physical compute buildout and compute-dependent algorithmic progress. → Different: a system a step above us routes around constraints we think are binding.

General form: any base rate or analogy is dissolved by positing a future system of sufficient magnitude that the comparison breaks.

Thesis

The load-bearing premise of AI x-risk arguments is “AI is different.” Therefore, the strength of it merits some specific investigation.

Here, I’d argue that assuming this premise as strongly as is often done is epistemically fraught:

1. This prior is philosophical, not empirical.

2. Philosophical arguments of this kind predict the world badly.

3. Because the premise is self-sealing, it — rather than any fact — drives the enormous spread in p(risk) across well-informed people.

Part 1 — philosophical, not empirical.

Each rebuttal above is an a priori claim about what sufficient intelligence entails (optimization power, workaround-finding), derived from a concept of intelligence, not from observed instances. Tellingly, current AI is exempted from historical comparisons in a way we wouldn’t be tempted to do for a different change like the internet, a political event, or social media. That concession relocates the claim from the empirical register (where base rates run against it) to the conceptual/future register (where no data can reach it).

The self-sealing works through three moves: reference-class escape (the object is “superintelligence,” outside any sample); capability-as-universal-solvent (any bottleneck dissolves under enough intelligence); disanalogy on demand (the system is underspecified enough to differ in whatever way the argument needs). The premise cannot lose — skeptics’ evidence slides off, believers’ scenarios are never refuted by present systems. A prior that no observation can move is exactly what produces 0.01-vs-0.5 splits among people sharing the same facts.

Part 2 — it proves too much.

Outside view.

Longquoting Dwarkesh:

• “I’ve been reading The House of Government recently. It’s a fascinating account of people involved in the Russian Revolution. There were many different factions of people who were disillusioned with the Czarist regime - the anarchists, the Mensheviks, Bolsheviks, the social revolutionaries, the Decembrists. They intensely debated the dichotomies which were most salient to them given their milieu.

  • The “decisive battle” … covered all the usual points of disagreement: the “working class” versus “the people”; the “sober calculation” versus “great deeds and self-sacrifice”; “objectivism” versus “subjectivism”; and “universal laws of development” versus “Russia’s uniqueness.”

• Yet none of them anticipated the considerations we now recognize to be far more relevant to economic development: dispersed knowledge, voluntary exchange, and entrepreneurial innovation.

• I think about this whenever my Bay Area friends debate AGI - will there be a software-only singularity, adversarial misalignment, training gaming, explosive growth, etc, etc? Maybe the frameworks we’re using and the questions we’re asking are fundamentally misguided.

• Given this topic is so epistemically murky that someone smart can come up with a new consideration that alters your key conclusions, how much should you update on the most recent compelling story you’ve heard?”

Here is an even stronger, more deductive presentation of this argument, the predict–postdict gap:

We still can’t agree on the causes of events that already happened with full archives — how much the internet contributed to GDP, what actually ended slavery. If retrodiction from extensive factual knowledge fails, prediction of an unprecedented system from armchair deductive argument should fail worse.

Arguments of this form have a poor forecasting record; “AI is different” is one. As the crux of many AI safety arguments, it’s important to have strong reasons to believe it to overcome the above.