70 Million TB/s: The Three-Lever Mechanism Driving AI’s Memory Bandwidth Growth

NVIDIA‘s B200 GPU delivers 8 TB/s of HBM3e memory bandwidth per chip. A DGX B200 system with eight GPUs delivers 64 TB/s. A rack of four DGX systems approaches 256 TB/s. A full-scale training cluster with hundreds of racks exceeds 70 million TB/s of aggregate memory bandwidth. That number sounds abstract until you understand what it means for AI model training and inference: for the decode phase of inference, generating tokens one at a time, memory bandwidth rather than compute FLOPS is typically the binding constraint on throughput. Prefill (processing the input prompt) and training are generally more compute-bound. The AI hardware race in 2026 is not only about who has the most transistors. For a large and growing share of real-world workloads, dominated by decode-phase inference, it is about who can move data to those transistors fastest.

The three levers of AI hardware performance are compute (measured in FLOPS, how many operations per second), memory bandwidth (measured in TB/s, how fast data can be fed to the compute units), and interconnect (measured in GB/s per link, how fast GPUs can communicate with each other during distributed training). Every AI hardware generation improves all three. But the relative importance of each lever has shifted, and shifts differently depending on the workload phase. In 2020, compute was the binding constraint across most workloads: models needed more FLOPS than hardware could provide. By 2026, compute has scaled faster than memory bandwidth, creating a new bottleneck that is most acute specifically during memory-bound phases like decode-heavy inference, less so during compute-bound phases like prefill and training.

Why Memory Bandwidth Is the Bottleneck for Decode

A modern frontier model (GPT-5 class, 1 trillion+ parameters) stores its parameters in GPU memory (HBM). During the decode phase of inference, every token generated requires reading a significant portion of those parameters from memory and feeding them to the compute units, one token at a time. The compute units can process the data faster than the memory system can deliver it for this access pattern. The GPU’s arithmetic logic units sit comparatively idle, waiting for data. This is the “memory wall” problem, and it determines the maximum tokens-per-second throughput specifically for decode-heavy inference workloads; prefill, where the model processes many tokens of prompt at once in parallel, keeps compute units substantially busier and is comparatively less memory-bound.

The math is straightforward for the decode case. A 1 trillion parameter model stored in FP16 requires 2 TB of memory. Generating one token requires reading a fraction of those parameters (determined by the model architecture and batch size). At 8 TB/s memory bandwidth (B200), a single GPU can read its entire local memory in roughly 125 milliseconds. For models that exceed single-GPU memory capacity (which all frontier models do), the parameters are split across multiple GPUs, and the interconnect bandwidth determines how fast the split model can synchronize. For decode-phase inference specifically, the pipeline from “user sends a query” to “model generates the next token” is gated by how fast data moves through memory and across interconnects more than by how fast the compute units can process it once the data arrives.

The Three-Lever Mechanism

What This Means for AI Cost Structure

The memory bandwidth bottleneck directly affects AI inference economics, specifically for decode-heavy workloads. Inference cost for those workloads is determined largely by how many tokens per second a GPU can generate, which is limited by memory bandwidth rather than compute for decode. A GPU with 2x the compute but the same memory bandwidth generates tokens at roughly the same speed for memory-bound decode workloads, though it can still meaningfully speed up compute-bound prefill. This is part of why NVIDIA’s Blackwell generation invested heavily in HBM3e memory (8 TB/s vs 3.35 TB/s) alongside its compute gains. The compute improvement matters more for training and prefill. The memory improvement matters more for decode-heavy inference. And inference makes up a large and growing share of enterprise AI spending in 2026.

Google’s TurboQuant is a KV-cache compression method, not a general model-weight quantization technique: it targets the key-value cache that grows with context length during inference, compressing it toward 3-4 bits per value to reduce the memory footprint that specifically constrains decode-phase throughput and concurrent-user capacity. This is a narrower, different problem from quantizing the model’s weights themselves, though both are examples of the same broader pattern: if you cannot increase memory bandwidth, reduce the amount of data that needs to flow through it. Quantization of model weights (reducing them from FP16 to INT4 or lower), KV-cache compression like TurboQuant, speculative decoding, and mixture-of-experts routing are all, in their own specific ways, techniques for reducing memory bandwidth or memory footprint requirements during the memory-bound phases of inference.

The HBM Supply Chain

HBM (High Bandwidth Memory) is manufactured by three companies: SK Hynix (South Korea), Samsung (South Korea), and Micron (United States). SK Hynix holds approximately 50% market share for HBM3e, the current generation. Samsung and Micron split the remainder. HBM production requires advanced packaging technology (stacking multiple DRAM dies with through-silicon vias) that is capacity-constrained. The demand for HBM from AI GPU manufacturers exceeds current production capacity, which is why GPU delivery timelines extend 6 to 12 months and why GPU prices remain elevated despite increasing production volumes.

The HBM supply constraint is the hidden bottleneck in the AI hardware supply chain. NVIDIA can design faster GPUs. TSMC can fabricate the GPU chips. But the complete GPU cannot ship without HBM, and HBM production scales more slowly than GPU demand. This constraint explains why NVIDIA’s data center revenue growth (while massive) is supply-constrained rather than demand-constrained. The company sells every GPU it can produce. The limit is how many GPUs it can produce, which is partially determined by HBM availability.

What Comes After the Memory Wall

The industry’s response to the memory wall operates on three timescales. In the near term (2026 to 2027), algorithmic optimizations (quantization, sparsity, KV-cache optimization) reduce memory bandwidth requirements for memory-bound phases without changing hardware. In the medium term (2027 to 2029), next-generation memory technologies (HBM4, with projected 2x bandwidth improvement over HBM3e) and compute-near-memory architectures (placing processing elements directly in the memory stack) attack the problem at the hardware level. In the long term (2029+), fundamentally new computing architectures (optical interconnects, photonic computing, neuromorphic chips) may reduce the memory wall’s impact by changing how compute and memory interact.

For AI builders in 2026, the memory bandwidth constraint has immediate practical implications, particularly for decode-heavy inference workloads. Inference cost per token for those workloads is determined substantially by memory bandwidth utilization, not compute utilization. Optimizing that class of inference means optimizing memory access patterns. For memory-bound decode workloads specifically, the cheapest way to reduce inference costs is often not to buy more GPUs but to reduce the memory bandwidth each inference request consumes through quantization, batching, and caching. The companies that understand this distinction, and apply it to the right workload phase, are positioned to run inference more efficiently than those applying a one-size-fits-all optimization strategy.

Sources: NVIDIA Blackwell architecture white paper (B200 specifications); NVIDIA DGX B200 system specifications; AMD MI300X technical specifications; Google TPU v5e documentation; SK Hynix HBM3e production data; Samsung/Micron HBM market share (TrendForce); Google TurboQuant technical blog; AnalyticsWeek inference economics analysis; NVIDIA GTC 2026 presentations. Updated 2026-08-18: qualified “memory bandwidth is the bottleneck” to the decode phase of inference specifically, distinguishing it from the more compute-bound prefill and training phases, and corrected the description of TurboQuant from a general model-weight quantization technique to what it actually is, a KV-cache compression method.