NVIDIA Nemotron 3 Super: The Open-Weight Model That Beats GPT-4 on Code

NVIDIA released Nemotron 3 Super on March 12, 2026, a 120-billion-parameter open-weight model with 12 billion active parameters per token. The model uses a hybrid Mamba-Transformer mixture-of-experts architecture with a 1-million-token context window. It is available on Hugging Face under NVIDIA’s Open Model License with full weights, training datasets (10 trillion tokens), and reinforcement learning recipes. NVIDIA claims 2.2x higher inference throughput than OpenAI‘s GPT-OSS-120B and 7.5x higher throughput than Alibaba’s Qwen3.5-122B on the 8k-input/16k-output benchmark setting.

As of March 12, 2026, the model ranked first among open-weight models on the SWE-Bench Verified leaderboard (swebench.com) with 60.47%, placing it competitive with closed frontier models on code generation. It also ranked first on DeepResearch Bench (NVIDIA’s multi-step research evaluation, March 2026) and first in its efficiency class on Artificial Analysis (as of publication date). Leaderboard rankings are dynamic; the positions listed here reflect the state at time of writing. But the real story is not the benchmarks. It is why NVIDIA is spending $26 billion over five years to give away frontier AI models for free.

The Three-Architecture Hybrid That Makes It Work

Nemotron 3 Super combines three distinct architectural components in a way no other production model does. The backbone alternates between Mamba 2 state-space layers (which process sequences in linear time, making million-token contexts tractable) and Transformer attention layers (which provide the reasoning precision that pure state-space models lack). On top of this hybrid backbone sits a Latent Mixture-of-Experts layer that compresses token representations before routing them to specialist expert networks.

The Latent MoE design is the architectural differentiator. Standard MoE models route full token embeddings to expert networks. Nemotron’s approach compresses the token into a latent representation first, then routes the compressed form. This allows the model to activate 4x as many expert specialists for the same inference cost, because each expert processes a smaller input. The result: 120 billion total parameters but only 12 billion active per forward pass. That 10:1 ratio between total and active parameters is aggressive even by MoE standards.

Native NVFP4: Training in 4-Bit From Day One

Most quantized models start as full-precision (FP32 or BF16) models and get compressed to lower precision after training. That post-training quantization always introduces accuracy loss. Nemotron 3 Super takes a different approach: the majority of multiply-accumulate operations during pretraining run in NVFP4, NVIDIA’s 4-bit floating-point format optimized for Blackwell GPUs. The model learns to be accurate within 4-bit constraints from the first gradient update.

The practical impact: on Blackwell B200 GPUs, Nemotron 3 Super runs 4x faster than FP8 models on the previous Hopper H100 architecture. On H100s, it still outperforms competing open models because the native FP4 training means quantization artifacts are minimal. This is not an afterthought optimization. It is an architecture decision that ties Nemotron’s best performance to NVIDIA’s latest hardware.

Multi-Token Prediction and Speculative Decoding

Standard language models predict one token at a time. Nemotron 3 Super’s MTP (Multi-Token Prediction) heads predict multiple future tokens in a single forward pass. This enables native speculative decoding without a separate draft model. The MTP heads share weights with the main model, which means speculative drafts stay consistent even at longer draft lengths, where independently trained draft models typically degrade.

For code generation and structured output tasks (tool calls, JSON, API responses), MTP delivers up to 3x wall-clock speedups. For agentic workflows where every tool call, reasoning step, and context slice gets re-processed, this speed improvement compounds across multi-step chains. NVIDIA’s pitch to enterprise is direct: agentic AI systems generate 15x more tokens than standard chat. If your inference costs scale linearly with token count, a 3x speedup on generation is a 3x reduction in the per-task cost of running agents.

Why NVIDIA Gives Away Frontier Models

A 2025 financial filing revealed NVIDIA plans to spend $26 billion over five years building open-weight AI models. Bryan Catanzaro, VP of applied deep learning research, confirmed to Wired that the company recently finished pretraining a 550-billion-parameter model (Nemotron 3 Ultra, not yet released). This is not philanthropy. NVIDIA’s business model is selling GPUs. Models tuned for NVIDIA hardware create a software lock-in layer: if your production model runs fastest on Blackwell because it was pretrained in NVFP4, you buy Blackwell GPUs. Open weights make the model free. The hardware to run it at peak efficiency is not.

The competitive dynamic is equally clear. Alibaba’s Qwen, Meta’s Llama, and Google DeepMind‘s Gemma are all open-weight model families that can run on any hardware. NVIDIA releasing competitive open models that perform best on its own GPUs is a defensive play to prevent customers from optimizing their inference stacks for AMD MI300X or Google TPUs. Perplexity, Palantir, Cadence, and Siemens are already integrating Nemotron 3 Super into production workflows.

What the Benchmarks Show (and What They Miss)

The open-weight model tier is no longer a consolation prize. Nemotron 3 Super, combined with Qwen 3.5 and Llama 4, means enterprise teams can run competitive AI agents on their own infrastructure without API dependencies. The question is no longer whether open models can match closed ones on specific tasks. They can. The question is whether the operational complexity of self-hosting outweighs the control and cost advantages. For NVIDIA, the answer to that question does not matter, because they sell the hardware either way.

Sources: NVIDIA technical blog, March 2026; NVIDIA Newsroom GTC announcement; Hugging Face model card; VentureBeat analysis; Dataconomy coverage; The New Stack; NVIDIA Open Model License.

The Training Data Release Changes the Game

NVIDIA did not just release weights. It published 10 trillion curated pretraining tokens, 40 million post-training alignment samples, and the complete reinforcement learning recipe across 21 environment configurations using NeMo Gym (1.2 million environment rollouts for tool-calling and planning verification). This is the most complete training pipeline disclosure from any major AI lab for a model of this scale. Competitors release weights. NVIDIA released the recipe.

For research teams, the training recipe is more valuable than the model itself. Weights are a snapshot. Recipes are reproducible. Any team with sufficient compute can retrain or modify the pipeline for their domain. The specialized pretraining datasets cover code concepts, algorithms, formal logic, economics, and structured reasoning. NVIDIA is building an ecosystem where the best path to a production-ready agent model starts with Nemotron’s pipeline running on NVIDIA hardware. The model is the lure. The hardware dependency is the business.

Nemotron 3 Ultra (approximately 500 billion parameters, 50 billion active) has been confirmed by NVIDIA executives but has no release date. If the Super model’s architectural pattern scales to Ultra, the open-weight model tier gets a genuine frontier-class entrant in the second half of 2026. That would force every AI company selling API access to justify pricing against a free, self-hostable alternative. The margin compression in AI inference is coming, and NVIDIA is engineering it deliberately.

Updated 2026-08-18: Date-stamped all leaderboard superlatives to March 12, 2026. Named the specific eval harnesses: SWE-Bench Verified (swebench.com), DeepResearch Bench (NVIDIA internal), Artificial Analysis, PinchBench. Added note that leaderboard rankings change as new models submit.