On July 16, 2026, Moonshot AI released Kimi K3, a 2.8-trillion-parameter (2.8T) Mixture-of-Experts model — the largest open-weight model in the world as of its release. The API and chat went live on launch day; the full weights are scheduled for July 27.
This article covers K3 from five angles: architecture, new techniques, serving infrastructure, performance, and comparison. Because the release is so recent, much of the information is Moonshot’s own announcement or third-party interpretation. I distinguish official claims from independent verification and flag uncertain items. Once the formal technical report and weights publish (targeted July 27), parts of this picture may need updating.
At a Glance
| Item | Value | Source type |
|---|---|---|
| Total parameters | 2.8T | Official |
| MoE config | 16 of 896 experts active per token | Official |
| Active parameters | ~50B | Third-party estimate (not officially stated) |
| Context | 1,048,576 tokens (1M) | Official |
| Modality | Text + image + video (native) | Official |
| Reasoning | Always-on “thinking mode” | Official |
| Precision | MXFP4 weights + MXFP8 activations (QAT) | Official |
| Release | API 7/16, weights 7/27 | Official |
The ~50B active-parameter figure is an estimate based on 16/896 × 2.8T. Because of LatentMoE’s shared parameters, it may not be exactly this value, and it is not a figure Moonshot explicitly stated.
Architecture
K3’s structure comes down to three in-house techniques.
Kimi Delta Attention (KDA)
Handling a 1M-token context is bottlenecked by attention’s quadratic cost. K3 eases this with KDA, a hybrid linear attention — some layers use KDA, some use full attention.
The core mechanism is a variant of Gated DeltaNet (per the most detailed third-party analysis). Standard DeltaNet multiplies the previous recurrent state by a single scalar decay, meaning every hidden dimension forgets at the same rate. KDA replaces that scalar with a per-channel vector turned into a diagonal matrix, so each dimension gets its own forgetting rate. Moonshot open-sourced FlashKDA kernels.
Moonshot’s claimed gains are up to 6.3× decoding at 1M context, up to 75% KV-cache reduction, and 1.72–2.22× prefill on H20 GPUs. These are Moonshot’s own figures with no independent measurement yet.
Deep dive: Kimi Delta Attention paper review (Kimi Linear, arXiv:2510.26692)
Attention Residuals (AttnRes)
A replacement for residual connections. Instead of each layer accumulating every previous layer’s state identically, a layer acts like a query that looks back over earlier layers and pulls in only the representations it needs with learned weights. Moonshot claims about 25% higher training efficiency at under 2% additional cost.
Deep dive: Attention Residuals paper review (arXiv:2603.15031)
Stable LatentMoE
The MoE routing framework. Its notable element is Quantile Balancing: a token routes to an expert if its router score lands in the top quantile — deterministic, hyperparameter-free, and claimed to guarantee even utilization with zero dead experts, replacing heuristic aux-loss balancing. Also mentioned are a Per-Head Muon optimizer, a SiTU activation, and Gated MLA.
Deep dive: LatentMoE paper review (NVIDIA, arXiv:2601.18089; K3’s Stable LatentMoE details unpublished)
Combining the three, Moonshot claims about 2.5× scaling efficiency over K2.
Serving Infrastructure
Precision and Memory
K3’s weights are MXFP4 (4-bit float with per-block scaling), and its activations are MXFP8. Notably, this is not post-training quantization but quantization-aware training (QAT) from the SFT stage onward — the model is trained to be resilient to the low-precision format it is served in. MXFP4 is natively supported on NVIDIA Blackwell and AMD MI400.
The memory math is simple: bytes per parameter times the number of parameters. K3 stores its 2.8T parameters in MXFP4 (4 bits = 0.5 bytes per parameter), so the weights alone need 2.8T × 0.5 bytes = about 1.4 TB.
| Precision | Per parameter | 2.8T weights |
|---|---|---|
| FP16 | 2 bytes | 5.6 TB |
| FP8 | 1 byte | 2.8 TB |
| MXFP4 | 0.5 bytes | 1.4 TB |
The 594 GB figure cited in some sources does not match 2.8T at any standard precision (2.8T at 4-bit is 1.4 TB; 594 GB would imply ~1.7 bits per parameter, which does not hold). It appears to be confused with a different model or configuration, and I disregard it for K3’s full weights.
Since the weights alone are 1.4 TB, even on 80 GB accelerators that is 18+ just to hold the weights. Adding the KV cache for a 1M context and activations gives Moonshot’s recommended supernode of 64+ accelerators.
Serving Stack and Pricing
- Inference stacks: vLLM (Moonshot contributed the KDA and prefill-cache implementation), SGLang, TensorRT-LLM, Transformers, KTransformers, xLLM. Moonshot itself serves it on Mooncake disaggregated inference.
- Pricing (Moonshot): $3 per 1M input tokens (cache miss), $0.30 cache hit, $15 output. The OpenRouter listing is $3/$15 with no cache discount.
- Price positioning: at $3/$15, roughly 60% of GPT-5.6 Sol’s ($5/$30). It places capability just below the frontier while pricing tokens below it. (This is a per-token price comparison, not a capability tier.)
Open Weights and License
The full weights are scheduled for July 27. Some outlets reported the license as “Modified MIT,” but the HuggingFace community overview says it is “TBD with the weight release,” and Moonshot’s official docs do not state a license. I treat it as unconfirmed. As of research time, no K3 repo yet exists on huggingface.co/moonshotai.
Training Hardware
Moonshot did not disclose training hardware. K3’s benchmark docs reference H200 and “a GPGPU from an alternative vendor” (interpreted by the press as Huawei Ascend), but this is unconfirmed. The “working around U.S. compute limits” narrative is press framing, not a documented Moonshot claim. I do not assert the training silicon as fact.
Performance
Independent Composite
The most trustworthy independent metric is Artificial Analysis’s Intelligence Index.
K3 scores 57 — top-tier, behind Fable 5 (60) and GPT-5.6 Sol (59), comparable to Opus 4.8 (56). Some reports said “#3 on the Intelligence Index,” but the same source’s body and other outlets place it around 4th (57). “Top-3 to top-4 tier, 57” is the accurate reading. On the independent agentic benchmark AA-Briefcase, it reached Elo 1547, a large jump from K2.6 (~815), just below Fable 5 (1583).
Strengths
- Frontend Code Arena (independent, blind developer evaluation): Elo 1679, #1, ahead of Fable 5 (1631). This is K3’s headline “beats a frontier model” evidence.
- SWE Marathon (sustained multi-hour engineering, Moonshot-reported): 42.0, ahead of GPT-5.6 Sol (39) and Opus 4.8 (40). GLM-5.2 collapses to 13.
- GPQA Diamond: 93.5%, the best open-weight result — though still below GPT-5.6 Sol (94.1%).
- Narrow wins on SpreadsheetBench 2 (34.8) and Automation Bench (30.8).
Weaknesses (Guarding Against Hype)
For balance, the weaknesses are just as clear.
- Humanity’s Last Exam: 43.5, the lowest of the frontier group (Fable 5 53.3, Opus 4.8 49.8, GPT-5.6 Sol 44.5). It trails on the hardest reasoning benchmark.
- Rising hallucination: on AA-Omniscience, accuracy rose from K2.6’s 33% to 46%, but the hallucination rate also rose from 39% to 51%. As it scaled up, it more often gets things confidently wrong.
- Losing its home benchmark: on its own Kimi Code Bench 2.0, it scores 72.9, losing to Fable 5 (76.9).
- Terminal-Bench 2.1 (88.3) narrowly loses to GPT-5.6 Sol (88.8).
Pitfalls in Reading the Benchmarks
Moonshot’s reported numbers come with methodology caveats.
- BrowseComp 91.2 is with context compaction triggered at 300K tokens. Without compaction, on the full 1M window, it drops to 90.4 — a tie with GPT-5.6 Sol. The “win” is configuration-dependent.
- Harness mismatch in coding benchmarks: Moonshot measured each model in a different harness (K3 in KimiCode, Fable 5 in Claude Code, GPT-5.6 Sol in Codex). Cross-model comparisons are not apples-to-apples.
No Speed Data
There is one important gap. Independent inference-speed data (tokens/sec, TTFT) does not yet exist. Artificial Analysis’s provider page shows output speed and latency as “no data available.” The only speed-adjacent statement is KDA’s architectural claim of “6.3× decoding at 1M,” which is not measured throughput. This article carries no unverifiable speed figures.
Comparison
Lineage: K2 → K3
| Kimi K2 (2025.7) | Kimi K3 (2026.7) | |
|---|---|---|
| Total parameters | 1T MoE | 2.8T MoE |
| Active/token | 32B | ~50B (est.) |
| Experts | 384 routed + 1 shared | 16 of 896 active |
| Attention | Multi-head Latent (MLA) | KDA + AttnRes, Gated MLA |
| Context | 128K | 1M |
| Modality | Text | Text + image + video |
| Reasoning | - | Always-on thinking mode |
In one year: 2.8× the parameters, 8× the context, plus multimodality and always-on reasoning. Moonshot’s narrative is that efficiency gains (~2.5×) underwrite this expansion.
Against Other Open-Weight Models
K3 is the largest open-weight model at release. Rough size comparison (third-party aggregation, approximate):
| Model | Total / active | Context |
|---|---|---|
| Kimi K3 | 2.8T / ~50B | 1M |
| DeepSeek V3.2 | 685B | 128K |
| Qwen3-Coder | 480B / 35B | - |
| GLM-5.2 | 744B / ~40B | - |
| Llama 4 Scout | - | 10M |
K3 broadly beats GLM-5.2 on published benchmarks (e.g., SWE Marathon 42.0 vs 13.0). I found no reliable head-to-head numbers against DeepSeek or Qwen, so I do not compare them.
Against Frontier Proprietary
Moonshot itself concedes that “overall performance still trails Fable 5 and GPT-5.6 Sol.” K3 wins isolated benchmarks (Frontend Arena, SWE Marathon, FrontierSWE, SpreadsheetBench) but loses the composites, HLE, DeepSWE, and its own KCB. Against Opus 4.8 it is roughly even or slightly ahead (AA Index 57 vs 56). On cost, it runs about $0.94/task versus GPT-5.6 Sol’s $1.04.
Takeaways
Kimi K3’s significance is twofold. First, an open-weight model has narrowed the gap at 2.8T scale to just below the frontier proprietary tier. Second, with in-house techniques like KDA, Attention Residuals, and Stable LatentMoE, it attempts efficiency-based scaling rather than brute size. In particular, training with QAT for MXFP4 squarely targets the serving economics of large models.
At the same time, hype deserves caution. It trails the frontier on composites, is weak on the hardest reasoning (HLE), and its hallucination rate actually rose. Some of the headline benchmark wins depend on compaction settings or harness differences. Independent speed data does not yet exist, and the license and training hardware are unconfirmed.
After July 27, when the formal technical report and weights publish, clearer information on KDA’s and AttnRes’s actual mechanisms, serving cost, and reproducible benchmarks should sharpen this picture. It will be worth updating this analysis then.
References
- Moonshot official blog: kimi.com/blog/kimi-k3
- Independent composite: Artificial Analysis — Kimi K3
- API/pricing: OpenRouter — moonshotai/kimi-k3
- Architecture analysis (third-party): Ken Huang, “Demystifying Kimi K3” · HuggingFace: MXFP4 overview
- Reporting: VentureBeat · Tom’s Hardware · the-decoder
Many figures are Moonshot’s own reporting, stated separately from independently verified items (Artificial Analysis, Frontend Code Arena). Unconfirmed items are flagged in the text.