Paper Info
- Title: LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts
- Authors: Venmugil Elango et al. (NVIDIA)
- arXiv: 2601.18089 (January 2026)
This post covers the paper behind the Stable LatentMoE mentioned in the Kimi K3 analysis. One thing to be clear up front: the LatentMoE paper is NVIDIA’s research, adopted in the Nemotron-3 Super/Ultra models. Moonshot’s K3 adapts this LatentMoE concept as “Stable LatentMoE.” Keeping that distinction, let’s start from the original paper.
The Problem: Is MoE Optimal per Cost?
MoE (Mixture of Experts) activates only a subset of experts per token, keeping total parameters large while per-token computation small. It is the standard way to scale large models.
The paper’s question: how optimal are existing MoE architectures with respect to inference cost? Cost is measured on two axes — accuracy per FLOP and accuracy per parameter. Is there room to reach higher accuracy at the same computation and the same parameters?
LatentMoE
The core idea is to decouple expert computation from the hidden dimension.
In standard MoE, experts operate directly in the model’s large hidden dimension d. LatentMoE first projects the incoming activations into a low-dimensional latent space d'. It performs routing and expert computation in this compressed latent space, then expands the result back to the original dimension d.
Decoupling routing and expert computation from the model’s hidden dimension is the key. Because experts operate in a small d', the same FLOPs and parameters can afford more experts or a more efficient representation.
Results
- Scale: validated at up to 95B parameters with 1T-token training
- Efficiency: consistently beats standard MoE on accuracy per FLOP and per parameter
- Adoption: applied in NVIDIA’s Nemotron-3 Super and Ultra models
The key point is beating standard MoE on cost-normalized metrics — accuracy per FLOP and per parameter. The direction is not scaling absolute size but reaching better accuracy at the same cost.
Limitations
From the abstract alone, details of the architecture, benchmark numbers, and load-balancing method are limited. How the choice of latent dimension d' and the added projection/expansion computation offset the gains needs the full paper to assess.
Kimi K3’s Stable LatentMoE
Kimi K3 adapts this LatentMoE concept under the name Stable LatentMoE, adding its own load-balancing method, Quantile Balancing. A token routes to an expert if its router score lands in the top quantile — deterministic, hyperparameter-free, and claimed to guarantee even utilization with zero dead experts.
That said, the exact mathematical details of K3’s Stable LatentMoE and Quantile Balancing have not yet been released. What is currently confirmed is that K3’s 896-expert structure stands on this direction. Once the formal technical report publishes, the difference between NVIDIA’s LatentMoE and Moonshot’s Stable LatentMoE should become clear.
References
- Venmugil Elango et al. (NVIDIA). “LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts.” arXiv:2601.18089, 2026. arXiv
- Related: Kimi K3 analysis · Kimi Delta Attention · Attention Residuals
The numbers are the paper’s reported values. The diagram is original, built from the paper’s concepts. K3’s Stable LatentMoE details are unpublished, so I did not speculate on them.