The headline number on Moonshot AI's Kimi K3, announced July 16, is 2.8 trillion parameters. The number worth studying is 6.3. That is the reported decode speedup — up to 6.3x faster — that a new attention mechanism called Kimi Delta Attention (KDA) claims to deliver at million-token context lengths. If it holds up when the weights ship, it is a concrete answer to the oldest cost problem in the transformer, and it matters more than the parameter count on the box.
One caveat has to sit at the front of everything that follows. As of this writing, around July 18, the full open weights and the complete technical report are not public. They are scheduled for July 27 on Moonshot's Hugging Face org, under a modified MIT license whose final terms ship with the files. Until then, K3 is API-only, and the architectural claims below come from Moonshot's announcement and preliminary materials — not a peer-reviewable paper. I will flag throughout which claims are Moonshot's own and which come from independent third parties, because on architecture the two are not yet the same thing.
Why a million-token window is a real number, not a spec-sheet number
Start with the problem KDA is built to attack, because the claim is meaningless without it.
Standard self-attention — the mechanism at the core of every transformer since 2017 — compares every token in the sequence with every other token. For a sequence of length n, that is roughly n² pairwise interactions to compute, plus a key-value cache that grows linearly but must be re-read at every step. The quadratic term is the wall. Double the context and you roughly quadruple the attention compute. Go from a 128K window to a 1M window — about 8x longer — and the attention cost scales by something like 8², or 64x.
That single fact explains most of the economics of long context. It is why long documents cost more than short ones by more than their length suggests, why context windows crept upward slowly for years, and why a genuine 1M-token window is an engineering achievement rather than a configuration flag. You do not get there by renting more memory. You get there by changing the shape of the cost curve, which means changing attention itself. This is the terrain I mapped in Beyond Transformers: The Architectures Trying to Escape the Context Window — the linear-attention and state-space line of work that has spent years trying to replace that n² with something closer to n.
What KDA reportedly is
Linear attention is the standard escape route from the quadratic wall. Instead of storing and re-scanning the full history of keys and values, a linear-attention layer maintains a fixed-size recurrent state that it updates token by token. Cost then scales roughly linearly with sequence length, and decoding a long context stops getting quadratically more expensive as the context grows. The catch, historically, is capability: full attention's power comes precisely from its ability to reach back and retrieve a specific earlier token by content, and compressing all of history into a fixed-size state tends to blur exactly that.
Moonshot describes KDA as a hybrid linear attention. The word carries the whole argument. The claim is that KDA keeps the linear/recurrent efficiency for the bulk of the work while retaining enough full-attention-style content-based retrieval to stay capable — and that this combination is what yields the reported up-to-6.3x faster decoding in million-token contexts. That is a coherent design target. Hybrid schemes that interleave cheap linear layers with a smaller number of full-attention layers, or that route some retrieval through a mechanism that can still address specific past tokens, are one of the more promising directions in the literature. If KDA is a well-executed instance of that idea at 2.8-trillion-parameter scale, it is a meaningful result.
Be precise about what "6.3x faster decoding" is and is not. It is a decode-throughput claim at long context — the speed of generating tokens when the context is already very large. It is Moonshot's own figure. It is not, on its own, a statement about quality, and it is not yet independently reproduced. Reserve judgment on the number until it can be measured against the released weights.
The rest of the efficiency stack
KDA is the load-bearing idea, but it sits inside a stack of efficiency choices worth naming because they compound.
Attention Residuals (AttnRes). Moonshot's second headline innovation "selectively retrieves representations across depth rather than accumulating them uniformly," which it credits with roughly 25% higher training efficiency at under 2% additional cost. Note the domain: this is a training-efficiency claim, not an inference one. The standard residual stream accumulates contributions from every layer uniformly; AttnRes reportedly lets later layers pull from earlier representations more selectively. A ~25% training-efficiency gain for under 2% overhead is the kind of asymmetry that, if real, changes what a given compute budget buys — but it is again self-reported, awaiting the technical report.
MoE sparsity. K3 is a sparse Mixture-of-Experts model: 896 total experts, 16 active per token. For any given token, only about 1.8% of the expert capacity fires. The 2.8-trillion-parameter total is mostly dormant at inference — the headline number is the size of the library, not the size of the read at any moment. This is why "largest open-weight model shipped to date" and "runnable at competitive cost" are not in tension. Moonshot lists further machinery around this — Stable LatentMoE with Quantile Balancing, Per-Head Muon optimization, a Sigmoid Tanh Unit activation, Gated MLA — none of which we can evaluate before the report, but all pointing at the same objective: keep the active compute per token small.
MXFP4 quantization. Weights in MXFP4 with MXFP8 activations push the memory and bandwidth cost per parameter down further. Four-bit weight formats are how a model this large stays servable at all.
Put the pieces together and the design has one consistent thesis: attack cost at every layer of the stack — the attention mechanism, the depth-wise information flow, the expert routing, and the numeric precision — rather than buying capability with raw dense scale.
| Component | What it targets | Reported effect | Source |
|---|---|---|---|
| Kimi Delta Attention (KDA) | Quadratic attention at long context | Up to 6.3x faster decoding at 1M tokens | Moonshot |
| Attention Residuals (AttnRes) | Depth-wise information flow | ~25% higher training efficiency, <2% added cost | Moonshot |
| MoE sparsity | Active compute per token | 16 of 896 experts active (~1.8%) | Moonshot |
| MXFP4 / MXFP8 | Memory and bandwidth per parameter | 4-bit weights, 8-bit activations | Moonshot |
| Overall | Scaling efficiency | ~2.5x better than Kimi K2; 21% fewer output tokens than K2.6 | Moonshot |
What it does to the cost curve
The efficiency argument shows up in the price, and here the arithmetic is worth doing.
K3's API prices input at $3.00 per million tokens and output at $15.00 per million. But cached input is $0.30 per million — a 10x discount — and Moonshot reports above 90% cache hits in coding workloads. If 90% of input tokens hit cache, the effective input rate is roughly 0.9 × $0.30 + 0.1 × $3.00 = $0.57 per million, not $3.00. For agentic coding, where the same repository context is re-sent across many turns, that gap is the difference between a workflow that pencils out and one that does not. Moonshot puts the net result at roughly half the per-task cost of Anthropic's Opus 4.8 — helped further by K3 reportedly using about 21% fewer output tokens than K2.6 on equivalent tasks, since output is the expensive side of the ledger.
This is the part founders should internalize. A cheap million-token context is not a bragging right; it is a change in which applications are viable. If long-context inference stops scaling quadratically in practice, whole classes of workload — full-repository reasoning, long-horizon agents, document-heavy retrieval without aggressive chunking — move from "technically possible, economically painful" to "default." KDA, if it delivers, is the mechanism underneath that shift.
The skepticism this deserves
I have written the optimistic reading. Now the discipline.
Hybrid-linear-attention schemes have a specific failure history: they frequently look excellent on decode speed while quietly trading away some long-range retrieval capability — the ability to pull one exact fact from deep in the context. Speed on a million-token window means little if fidelity at the far end of that window degrades. The honest test of KDA is not the 6.3x figure; it is whether K3 retains full-attention-grade retrieval across the whole 1M-token span, measured on tasks that actually stress long-range dependency rather than throughput. That measurement requires the weights, which is exactly why July 27 matters more than July 16.
The benchmark picture demands the same care, and here the source labels do real work. Independently, Artificial Analysis places K3 at a composite Elo of 1,547 — behind only Claude Fable 5, ahead of GPT-5.5, a 732-point jump from Kimi K2.6 — and Arena.AI's Frontend Code Arena ranks it #1 at 1,679, ahead of Claude Fable 5 and GPT-5.6 Sol. Those are third-party. Moonshot's own reporting is more measured: K3 "mostly beats Claude Opus 4.8 max and GPT-5.5 high" but trails Claude Fable 5 and GPT-5.6 Sol, with self-reported leads on Program Bench (77.8), SWE Marathon (42.0), BrowseComp (91.2), Automation Bench (30.8), and OmniDocBench (91.1). Note the apparent contradiction that isn't one: K3 sits at #1 on the narrow frontend-code board and #2 on the broad composite. Different benchmarks measuring different things — both can be true at once, and reading either as "K3 is the best model" overstates it.
One more claim needs a hard label. Moonshot's materials describe K3, over 48 hours of continuous autonomous agent operation, designing a physical chip to run a nano-scale version of itself — architectural design, optimization, and verification through the full pipeline using open-source EDA tools. Treat this as a company demonstration that has not been independently verified. It is a striking story and, until someone outside Moonshot reproduces it, only a story.
Why the mechanism, not the model, is the point
Step back to the market frame. The release drew comparisons to a "second DeepSeek shock," with AI-related stocks moving and commentators framing it as an open-weight model catching the closed US frontier at a fraction of the cost. Strip the drama and a sharper claim survives: if the frontier is now being contested partly on attention-mechanism efficiency rather than raw parameter scale, the set of organizations that can compete widens, and the cost curve everyone plans against bends downward.
That is the durable takeaway from K3, and it is an architectural one. The 2.8-trillion-parameter number invites you to think the race is still about size. KDA suggests the more interesting race is about the shape of the computation — where you can spend linearly instead of quadratically, and how much capability you can keep while you do it. That is also why the genuinely radical departures live one step further out: approaches that change what the model is representing rather than how it attends, the direction I traced in World Models: Why the Next Leap May Not Be a Bigger LLM. KDA is not that leap. It is a sharp, well-aimed optimization of the transformer we already have.
Which is exactly why it deserves the scrutiny of the whole field on July 27 — and exactly why the number to watch is not the one on the box.