Justy: So NVIDIA just dropped Nemotron 3 Ultra and I'm looking at the throughput numbers and thinking — okay, this is the move for anyone actually running agents in production right now.
Cody: Yeah. Five to six times faster inference than GLM-5.1 or Kimi while keeping accuracy flat. That's not a small delta.
Justy: Right, and the thing that got me is they're not hiding the tradeoff — it's 550 billion total parameters but only 55 billion active per token. Sparse by design.
Cody: Mixture of Experts, but they're using LatentMoE, which is the smarter routing pattern. Instead of routing at the token level — where you get load imbalance and expert collapse — they route in latent space. 2048-dimensional latent, 512 experts per layer, top-22 get activated.
Justy: Okay so that's less noisy than token-level routing.
Cody: Much less. You're routing a denser signal, so the router learns cleaner assignments. And then there's the hybrid Mamba-Attention thing — Mamba layers for the bulk of the token processing because they're linear complexity, way cheaper KV cache, and then selective attention layers where you actually need precision. Best of both.
Justy: So Mamba is handling the throughput and attention is handling the reasoning.
Cody: Exactly. The pattern is 108 layers total, and I think they're alternating or mixing them, but the point is you get the speed of Mamba — linear scaling with sequence length — without giving up the precision that transformers give you on hard reasoning steps.
Justy: And they pre-trained this thing on 20 trillion tokens, which is… a lot.
Cody: Two phases. First 15 trillion on diversity — broad domain coverage — then 5 trillion on high-quality data to refine. Pretty standard recipe at that scale now.
Justy: But here's the wild part: they trained it in NVFP4 the whole time. That's a four-bit mixed-precision format. Kept the final 16 layers and a few projection layers in BF16, but everything else was low precision. And they're saying this is the largest-scale stable NVFP4 training anyone's done.
Cody: Yeah, I saw the ablation. They branched off at 5T, 10T, and 16T tokens, switched to BF16, kept training for another 74 billion tokens, and the loss gap stayed below 0.4% on average. That's tight.
Justy: Right, so they're not leaving accuracy on the table by going low precision. The training is stable.
Cody: The trick is they use E2M1 datatype with two-dimensional block quantization on weights, Random Hadamard Transforms on the gradient inputs, and stochastic rounding. That's not trivial — you're managing numerical stability at every step.
Justy: Okay so that's the pre-training piece. But the post-training is where I think this gets interesting for agents specifically. They did SFT on curated data, then unified RL across like… reasoning, agentic, code, safety, usability, chat. Not single-domain.
Cody: Right.
Justy: And then — this is the move — they trained more than ten specialized teacher models using targeted recipes, and then ran Multi-teacher On-Policy Distillation to consolidate all of them into Ultra.
Cody: MOPD. So the teachers generate rollouts, and the student learns dense token-level corrections from ALL the teachers at once, not just one. That's how you avoid collapse to a single optimization target.
Justy: Exactly. Teacher diversity is the whole point. You've got agentic teachers, code teachers, reasoning teachers, all feeding signal into the same model.
Cody: And then they added reasoning effort control, so at inference time you can dial the compute spend up or down depending on the task complexity. That's actually useful for agent workloads where some steps are trivial and some are hard.
Justy: So you're not paying the same compute cost for every token in the agent loop.
Cody: No. You spend more on the hard reasoning steps, less on the routine ones. That's product-real for long-running agents.
Justy: And the throughput gains — they're claiming five to six times faster than GLM-5.1, Kimi, Qwen on 8K input, 64K output. That's a real benchmark.
Cody: Though fair note: they're measuring Nemotron with TRT-LLM and NVFP4 precision on GB200, and the other models with vLLM. So the comparison is a little tilted in Nemotron's favor.
Justy: Yeah, we always have to squint at those comparisons.
Cody: But even if you dial back the numbers by twenty or thirty percent, you're still looking at two to four times faster inference. That's not noise.
Justy: And they're open-sourcing the whole thing. Base model, post-trained, quantized checkpoints, training recipes, datasets — everything on HuggingFace. 173 billion tokens of fresh code data from GitHub through September 2025, synthetic legal datasets, post-training data, RL environments.
Cody: That's a real open release. Not just the weights — the infrastructure.
Justy: So who actually ships this? Teams running long-horizon agents where token latency compounds — code generation loops, research agents, anything that's hitting the model hundreds or thousands of times in a single task. The throughput gain means you finish faster or handle more concurrent requests on the same hardware.
Cody: Yeah. And the 1M token context is real infrastructure for agents — whole codebases in working memory, no constant re-briefing. That's the Nemotron 3 story from the start: context + speed = practical autonomous agents.
Justy: This is shippable. Not research-only.
Cody: Absolutely. The quantized checkpoints are ready to run. TRT-LLM integration is there. If you're building agent infrastructure, you can start today.
Justy: Okay so go grab the checkpoints from HuggingFace — search Nemotron 3 Ultra, there's the base model in BF16, the post-trained version, and the NVFP4 quantized model if you want to run on consumer hardware or tighter budgets. The training recipes are on GitHub under NVIDIA-NeMo, and all the datasets are there too if you want to fine-tune or extend.
Cody: And if you're already using TRT-LLM or vLLM, the integration is straightforward — it's just another HuggingFace checkpoint.
Justy: Yeah. This is one of those releases where the whole stack shows up at once. Pretty rare.