Nemotron has only 3B active parameters. So why doesn't it use as little memory as a 3B model?
Nemotron 3.5 Lightning uses around 3 billion parameters per token. But its VRAM footprint reflects a very different number. A look at what MoE architecture reduces, and what it doesn't.
A technical article NVIDIA published on September 15 included an interesting table.
Nemotron 3.5 Lightning has 30 billion parameters in total. Yet only around 3 billion of them are used to process each token. That does not mean the model occupies only 3 billion parameters' worth of memory.
NVIDIA's comparison table lists approximately 20 GB of memory for the 4-bit version. Why is that figure so high for a model with only 3B active parameters?
What MoE actually does
The answer starts with an accurate description of Mixture-of-Experts (MoE) architecture.
MoE does not shrink a model. It keeps the entire model in memory, but runs only a small portion of it when processing each token.
For Nemotron 3.5 Lightning, that entire model contains 30 billion parameters. Its MoE layers contain multiple expert networks. A component called the router selects which experts to use for the token being processed at each layer. The approximately 3B active parameters include shared model components as well as the selected experts. So this is not a fixed 3B model running at every step: the next token may activate different experts. All 30 billion parameters therefore need to remain somewhere the GPU can access them, ready for whichever experts are selected.
The space occupied by model weights depends primarily on the total parameter count and storage precision, such as BF16, FP8, or 4-bit. Total memory requirements during inference also include caches and other working buffers.
The comparison table in NVIDIA's article illustrates this. The figures below come directly from NVIDIA's table; they are not our own measurements:
| Model | Total / active parameters | Approximate BF16 memory | Approximate 4-bit memory |
|---|---|---|---|
| Nemotron 3.5 Lightning | 30B / 3B | 60 GB | 20 GB |
| Gemma 4 31B | 31B / 31B | 61 GB | 16 GB |
What stands out is that Nemotron and Gemma 4, with nearly identical total parameter counts, have broadly similar memory footprints despite one being MoE and the other dense. Even when the active parameter count is low, the total parameter count still accounts for much of the memory cost.
How not to read these numbers
A few points need clarification, because this table can easily invite misleading generalizations.
20 GB is not a fixed requirement. It is an approximate figure in NVIDIA's table. The source does not specify the detailed configuration behind it. The total parameter count explains why the memory requirement is not on the scale of a 3B model, but it does not, by itself, account for the exact 20 GB figure. Actual VRAM requirements vary with the quantization method, context window length, and runtime settings.
Not every expert network has to reside in VRAM. The table above assumes a model loaded entirely onto the GPU. Offloading methods that keep some weights in system memory and move them to the GPU as needed fall outside that scenario. They can reduce VRAM requirements, at the cost of speed.
It would be wrong to conclude that “MoE is faster, dense is smarter.” Nemotron shows high output speeds in the provider data NVIDIA compiled. However, these are not controlled comparisons using identical hardware and software configurations. We therefore cannot attribute the entire difference to MoE architecture or expect the same speed ratio on our own computers.
The provenance of the numbers matters, too. NVIDIA compiled the output-speed figures from Artificial Analysis results retrieved on August 31, 2026. The comparison covers only providers using NVIDIA GPUs. The article also notes that MoE's latency advantage narrows at high concurrency and that fine-tuning can lead to imbalanced routing across experts.
Why this matters
For anyone considering running a model locally, this distinction has a practical consequence: choosing hardware based on the “active parameters” figure can be misleading. An MoE model running quickly does not mean it has low memory requirements. If you want to run the model entirely on the GPU, you need to account for all its weights and the additional memory required during inference, not just the active portion. With partial loading, memory allocation and performance change.
Nemotron is a useful example for understanding what MoE is not: it does not shrink the model; it decides which part of that model runs at each step.
