Skip to content

You Don't Need an Intelligence Explosion for Self-Improving AI to Concentrate Power

Forget runaway superintelligence. Even a weak, bounded self-improvement loop compounds an advantage for whoever owns it, and the only thing standing between that and a monopoly is how fast capability diffuses.

By Mehdi8 min read
Share
On this page

Whether recursive self-improvement produces a runaway superintelligence is the wrong question to build a strategy around. The right one is quieter and almost certainly more decisive: even a weak, bounded loop, where AI modestly speeds up the work of building better AI, hands whoever owns that loop a compounding advantage, and compounding advantages concentrate. You do not need the explosion. You need only a persistent edge in the rate of improvement, run for enough cycles, and the arithmetic does the rest. The market-structure consequences of self-improving AI arrive long before, and independent of, anything that looks like a singularity.

Start by separating two claims that usually get welded together. I.J. Good named the strong one in 1965, in "Speculations Concerning the First Ultraintelligent Machine": once a machine can design machines better than we can, "there would then unquestionably be an intelligence explosion," the first ultraintelligent machine being "the last invention that man need ever make." Vinge and later Kurzweil packaged that into the singularity; Nick Bostrom, in Superintelligence, sorted the takeoff into fast, moderate, and slow trajectories and worried about goal-content integrity under self-modification, whether an agent rewriting itself preserves its objectives. All of that concerns the magnitude and safety of the loop. It is a real and open question, and I am setting it entirely aside, because the economic argument does not need any of it.

The loop is already real, and it is bounded

The weak version is not speculative. It describes what a frontier lab already does, and it has an exact precedent in computing: the self-hosting compiler. GCC compiles GCC. A better compiler produces a faster compiler, which compiles the next version faster, a genuine feedback loop that has run for decades and produced steady, compounding tooling gains without anything remotely explosive. Bootstrapping is recursive self-improvement in its mundane, load-bearing form. Nobody calls it a singularity because the returns are bounded: each turn of the crank helps, and the help does not diverge to infinity.

The AI version today is the same shape, still mild. Models help write and review the code for the next training run, generate and filter synthetic training data, propose and triage architecture and hyperparameter experiments, and write the CUDA kernels that make the next run cheaper. None of this is autonomous AI-designing-AI; it is a productivity multiplier on human research teams, and an honest read of the current regime is that the multiplier is real but modest, call it tens of percent on specific engineering tasks, not a rewriting of the whole R&D function. (That figure is my estimate of the present, not a measured result.) The loop exists. It is sub-critical. And sub-critical is enough.

The arithmetic of a small persistent edge

Here is the move most people miss because they are hunting for the explosion. A bounded loop that never diverges still concentrates, because the thing that compounds is not absolute capability. It is the relative rate of improvement between competitors.

Model it simply. Two labs are both improving, and each generation's model raises the productivity of building the next one, so capability grows by some percentage per development cycle. Let lab A improve 5% faster per cycle than lab B, not 5% better, 5% faster to improve, a small persistent edge in the rate. The gap between them after n cycles is (1.05)ⁿ:

Per-cycle edge 15 cycles 30 cycles 40 cycles
3% 1.6x 2.4x 3.3x
5% 2.1x 4.3x 7.0x
10% 4.2x 17x 45x

A 5% edge, the kind of difference that never surfaces in a headline benchmark, becomes a 4.3x capability gap over thirty cycles. That is the entire argument in one table. Compounding converts a persistent small advantage in rate into an extreme divergence in level, and it does so at improvement rates far too modest to call an explosion. The loop does not have to run away. It only has to run, and to run slightly better for one party than the others.

The obvious objection: why would the edge persist rather than wash out? Because the inputs to the loop are not randomly distributed. They are precisely the things that concentrate.

The inputs concentrate, and this is a scale economy, not a network effect

The loop has four inputs, and every one of them favors incumbents with scale.

Frontier compute. The single largest input, and the most concentrated. Leading-edge accelerators are supply-constrained, capital-rationed, and clustered among a handful of buyers who can commit years and billions ahead of need. Compute sets how many experiments you can run per cycle, and running more experiments per cycle is the improvement rate. Whoever has more of it gets a higher exponent.

Proprietary data. Not the open web, which everyone has, but the accumulated interaction data, human preference labels, and hard-won verified reasoning traces that a large deployed product generates and a challenger cannot simply download.

The engineering pipeline. The tacit, organizational capital: the training infrastructure, the evaluation harnesses, the reliability of a research org that has done many large runs. This does not transfer with a weights release.

The capital to run it all. Each cycle costs more than the last, and the ability to fund a losing quarter to buy a compounding edge is itself concentrated.

Notice what kind of advantage this is, because the label determines how it gets attacked. This is a supply-side scale-and-learning economy, not a demand-side network effect, and the difference is not pedantic. A network effect makes the product more valuable to each user as more other users join a shared graph; it saturates, clusters, and leaks, which is why it is the most over-claimed moat in startup strategy. The self-improvement loop is a different and, in this narrow case, sturdier animal: its advantage comes from owning scarce inputs to a compounding production process, not from users being reachable to each other. You do not attack it by winning a local cluster. You attack it by making its inputs cheap and its outputs common, which is exactly what the counter-force does.

Diffusion is the counter-force, and it has been faster than expected

If the story ended at compounding-plus-concentrated-inputs, the honest forecast would be a singleton, and I do not hold that forecast. The reason is a second dynamic pulling hard the other way: diffusion. Every generation the loop produces leaks, and it leaks fast, through two channels.

The first is open weights. When a lab releases a frontier-adjacent model openly, a Kimi-K3-style release, it does something specific to the compounding story: it hands everyone the current rung of the ladder. The leader's proprietary generation-N becomes the community's free generation-N a few months later, so the compounding that concentrated in one lab gets partially redistributed to every lab, every startup, and every researcher who can fine-tune. Open weights are a diffusion pump aimed directly at the leader's lead. They do not stop the loop; they compress the gap between the loop's owner and the field by resetting everyone's starting capability upward each cycle.

The second is the inference-cost collapse. The price of serving a fixed level of capability has been falling by roughly an order of magnitude a year, which means last year's frontier is this year's cheap commodity endpoint. I have argued that this single curve breaks most AI pricing models; here it does something more structural. The capability the loop produces diffuses down the cost curve almost as fast as the loop produces it. The leader can hold the frontier, but frontier-minus-one-generation becomes available to everyone at collapsing prices, so the economically useful gap, the capability others cannot cheaply match, stays narrow even as the raw capability gap widens.

Put plainly: compounding widens the distance between the leader and the pack; diffusion drags the pack forward each cycle. The market structure of AI is the outcome of that race.

My read: bet on a contested oligopoly, not a singleton

This is the analysis, labeled as such, a bet, not a deduction. The loop is real, it favors scale, and I take the concentration pressure seriously; anyone who waves it away because "the explosion didn't come" has misread which claim was load-bearing. But diffusion has repeatedly moved faster than the concentration story predicts. Open-weight releases have compressed leader-to-field gaps from years to months. The inference-cost collapse has been relentless and, if anything, has accelerated. Each time the leading lab has opened a gap, some combination of an open release and a cost collapse has pulled the field back within striking distance before the gap could compound into a moat.

So my bet is a contested oligopoly: a small number of scale players who each own a real, compounding loop and a real, persistent edge over everyone outside the group, but who cannot open a durable gap over each other, because diffusion keeps resetting the field faster than any one of them can compound away from the others. Not a singleton, because diffusion is too fast. Not a commodity free-for-all, because the input concentration is too real to fully democratize. A handful of players running the loop, a large fast-following field living one to two generations back at collapsing prices, and a leader's lead that is genuine but perpetually eroding, a metabolism, not a monument.

The thing that would break this bet is specific, and it is what to watch for: a regime where the loop's returns start to outrun diffusion. That would look like the improvement multiplier climbing steeply enough, AI meaningfully automating AI research rather than merely accelerating it, that a leader's next cycle ships and deploys before the last one has diffused. If the loop ever laps the diffusion cycle, compounding wins and the structure tips toward a singleton. I do not think we are there. I think it is the single most important thing to monitor, and it will show up in the rate first, not the headline capability.

The two variables that decide the market

If you want to track this rather than argue about it, ignore the benchmark leaderboards and watch two numbers, because they are the two sides of the race.

  1. Compute concentration — how much of frontier-compute supply pools among how few players, and whether that share is rising or spreading. This proxies the loop's exponent and how unevenly it is distributed. Rising concentration favors the singleton.

  2. Open-weight diffusion — the lag between a frontier release and a freely available model of comparable capability, and whether that lag is shrinking or growing. This, together with the inference-cost curve, proxies how fast the pack catches up. Shrinking lag favors the oligopoly, or something even more democratized.

The ratio of those two trends is the market structure of AI. Everything else, which lab is ahead this quarter, which benchmark got saturated, is noise on top of that signal.

The intelligence-explosion debate asks whether the machine runs away from us. The economic question is smaller and more immediate: whether the loop lets one owner run away from everyone else. Good's ultraintelligent machine may never arrive. The compounding does not wait for it, and neither should your read of who ends up owning the improvement loop.

Frequently asked questions

Does this argument depend on recursive self-improvement producing superintelligence?
No, and that's the point. The strategic consequences fall out of a weak, bounded loop where AI modestly speeds up building better AI. Compounding at even a few percent per cycle produces large divergence over enough cycles, so the concentration dynamic is present long before, and independent of, anything resembling an intelligence explosion. The strong-takeoff debate is a separate question from the market-structure one.
Why isn't the self-improvement loop just another network-effect moat?
Because it is a supply-side scale-and-learning economy, not a demand-side network effect. Its advantage comes from owning the inputs (frontier compute, proprietary data, the engineering pipeline, and the capital to run the loop), not from users being more valuable to each other on a shared graph. That distinction matters because the two decay differently: a network effect saturates and clusters, while a scale-and-learning loop is attacked mainly by input diffusion (open weights, cheaper inference) rather than by winning a local subgraph.
What are the two variables that actually decide the market structure?
Compute concentration and open-weight diffusion. The first measures how much of the loop's key input pools among a few players; the second measures how fast each generation's capability leaks to everyone. If compute concentrates faster than capability diffuses, the loop's owner pulls away toward a singleton. If diffusion keeps pace or wins, you get a contested oligopoly where the leader's edge is real but perpetually compressed. Track both, not the benchmark headlines.

Filed under Business & Strategy. How durable advantage is actually built — and lost.

Essays like this, in your inbox.

Thoughtful essays. No spam. Unsubscribe anytime.

Business & Strategy

Kimi K3 Turns "We Have the Best Model" Into a Dead Moat

Kimi K3 is a strategic event before it is a technical one: a near-frontier, open-weight model at roughly half Opus 4.8's per-task cost puts a ceiling on closed APIs and erases model access as a moat.

8 min read