A 70-billion-parameter model stored in the precision it was trained in takes about 140 gigabytes of memory just to hold its weights, before a single token of inference happens. Quantized to 8-bit integers, the same model fits in roughly 70 gigabytes. Quantized to 4-bit, it drops to around 35. That compression is not free, and the difference between a quantized model that works well and one that quietly gets worse at hard reasoning comes down to a handful of algorithms most users never see the name of.
Quantization is the general term for converting a model’s weights, and sometimes its activations, from the high-precision floating-point numbers they were trained in down to a smaller, discrete set of representable values. It is the single biggest lever for making a large model fit on smaller hardware, and it is also where a surprising amount of applied numerical analysis hides inside what looks like a simple rounding operation.
The Basic Operation
Every quantization scheme starts from the same core idea: pick a scaling factor, divide the original value by it, round to the nearest representable integer, and clip the result to whatever range the target bit-width allows. For a signed 4-bit integer, that range is negative eight to seven, giving sixteen distinct representable values total. An 8-bit integer allows 256 values. The scaling factor, computed from the actual distribution of weights being quantized, determines how that small set of integers gets stretched or compressed to cover the real range of values in the original tensor. Dequantizing later, multiplying the stored integer back by the scale, recovers an approximation of the original number, never the exact original, since rounding is lossy by construction.
The design choices inside that simple description turn out to matter enormously. Symmetric quantization centers the integer range on zero, which is simpler but wastes representational capacity if the real weight distribution is skewed. Asymmetric quantization adds a zero-point offset to better match a skewed distribution at the cost of slightly more computation during dequantization. Per-tensor quantization uses one scale for an entire weight matrix, which is fast but coarse. Per-channel or per-group quantization computes a separate scale for smaller slices, typically groups of 128 weights in current practice, which captures local variation in the weight distribution far better at a modest storage cost for the extra scale factors. Nearly every production quantization method in use today uses grouped, asymmetric quantization specifically because the accuracy difference against the coarser alternatives is large enough to matter.
Why Naive Rounding Breaks at Low Bit-Widths
Simple round-to-nearest quantization works reasonably well at 8-bit precision and gets noticeably worse at 4-bit, and the reason is specific rather than a vague statement about compression loss. Trained neural network weights are not uniformly distributed. A small number of weights, often well under one percent of the total, carry disproportionately large magnitudes and disproportionately large influence on the model’s output. At 8-bit precision there is enough resolution that these outlier weights still land close to their true value after rounding. At 4-bit precision, with only sixteen total representable values covering the entire range, an outlier weight either gets clipped, destroying it, or forces the scale factor to stretch wide enough to accommodate it, which then crushes the resolution available to every ordinary, non-outlier weight sharing that scale. Naive 4-bit quantization does not fail gradually. It tends to fail specifically on the small set of high-influence weights that a model relies on most, which is why generic rounding produces models that pass casual testing and then degrade sharply on complex reasoning and multi-step math.
The Two Algorithms That Fixed This
GPTQ, published by Elias Frantar and coauthors, solves the problem by quantizing a model layer by layer and using second-order information, an approximation of the Hessian, to decide how to round each weight so that the cumulative error introduced across the whole layer is minimized rather than treating each weight’s rounding error independently. It runs against a calibration dataset, commonly a sample of the C4 web text corpus, feeding real data through the model to observe how each layer actually behaves before deciding how to compress it. This calibration-driven, layer-by-layer correction is what separates GPTQ from naive rounding: it does not just round each number to its nearest low-bit value, it adjusts nearby weights to compensate for the error the rounding just introduced.
AWQ, Activation-aware Weight Quantization, takes a different angle on the same problem. Its starting observation is that the roughly one percent of weights that matter disproportionately can be identified in advance by looking at which weights interact with the largest-magnitude activations during a forward pass, not by analyzing the weights in isolation. Rather than trying to represent those salient weights at full precision inside an otherwise low-bit tensor, which complicates the storage format, AWQ applies a per-channel scaling transformation before quantization that effectively protects the salient weights by making their rounding error smaller, then quantizes the entire tensor uniformly afterward. The practical result is comparable or better accuracy than GPTQ at the same bit-width, with an approach that has become the default INT4 format for serving frameworks like vLLM specifically because the scaling transformation integrates cleanly into fast inference kernels. SmoothQuant approaches the same underlying tension, that activations are harder to quantize accurately than weights, from yet another direction, mathematically shifting quantization difficulty from activations onto the weights, where it is easier to correct for.
Where FP8 Fits
Integer quantization is not the only path. Floating-point formats with reduced bit-width, particularly FP8, keep a small exponent field that lets them represent both very large and very small numbers without the scale-factor gymnastics integer quantization requires, at the cost of fewer significant digits than a full FP16 or BF16 number. On current-generation hardware with native FP8 support, inference quality at FP8 is close enough to BF16 that it has become the practical default for teams that have that hardware available, which is exactly the precision DeepSeek built into V4’s KV cache and Lightning Indexer, storing most of the model at FP8 and pushing only the most sensitive components down to FP4. INT8 remains the more broadly compatible fallback for older hardware without native FP8 tensor cores, at roughly the same memory footprint but without FP8’s dynamic range advantage.
What the Numbers Actually Cost
Compression numbers alone undersell how workload-dependent the accuracy tradeoff really is. INT8 quantization is close to lossless for nearly every task teams test it against. INT4 with GPTQ or AWQ holds up well for summarization, classification, and routine code completion, and shows measurably more degradation on complex multi-step reasoning and difficult math, the same tasks where the small set of outlier weights protected by these algorithms matters most. This is not a flaw specific to any one method. It reflects the same underlying fact driving the whole field: a small fraction of a model’s weights carry a disproportionate share of its hardest capabilities, and any compression scheme has to work harder to protect exactly those weights than the rest of the network, the same lopsided-contribution pattern that shows up in which experts a router actually activates rather than which ones simply exist inside the model.
Deployment format adds a second, separate layer of tradeoffs on top of bit-width. GGUF, the format behind llama.cpp and Ollama, supports a range of quantization levels and, distinctively, enables splitting a model between GPU VRAM and system RAM, letting consumer hardware run models that would not otherwise fit in available video memory at the cost of the slower system-RAM portion dragging down overall throughput. That hybrid CPU and GPU execution path is a major reason GGUF-quantized models are so common outside dedicated inference infrastructure, even though a fully GPU-resident deployment at the same bit-width would run faster on hardware that could hold the whole model. ONNX takes a different, more standardized approach to the same problem, representing a quantized tensor and its scale factors as explicit graph operators rather than a bespoke file format, a tradeoff this site has covered separately. The same compression pressure shows up one layer up the stack too, in attempts to quantize the KV cache rather than the weights, where a technique that looks sound on paper can still fail once independent teams run it against production-scale traffic.
Where the Theory Runs Ahead of the Hardware
Quantization research has pushed well past what deployed hardware can actually accelerate, which is an honest limitation the compression numbers alone do not show. Six-bit quantization, sitting between the well-supported 4-bit and 8-bit tiers, has been shown in recent work to offer a meaningfully better accuracy-to-compression tradeoff than 4-bit at only a modest efficiency cost against 8-bit. It has also, by the same research’s own account, lacked native tensor core support on current GPUs, forcing implementations to emulate 6-bit arithmetic using higher-precision units and specialized kernels rather than getting a clean hardware speedup. Non-integer bit-widths, quantizing to something like 4.25 or 5.33 bits per weight by mixing precisions within a tensor, exist in research code today with no mainstream hardware path to accelerate them at all. The gap between what a quantization algorithm can theoretically achieve and what a GPU’s tensor cores actually know how to execute quickly is one of the more persistent, least discussed constraints on how fast this part of the field can move.
What Happens Next
The practical frontier right now is less about inventing new quantization math and more about closing the gap between algorithm and hardware: getting INT6 or similar intermediate precisions native tensor core support, and getting FP8 adoption to spread beyond the newest GPU generations so it stops being a privilege of teams with the latest hardware. On the algorithm side, calibration-dependent methods like GPTQ and AWQ still carry a subtler risk worth naming honestly: a model quantized against one calibration dataset’s distribution is being optimized to preserve accuracy specifically on inputs resembling that calibration set, and teams deploying a quantized model on a workload that looks meaningfully different from the calibration data are running an experiment they usually have not explicitly tested. The compression numbers on a model card describe the bit-width. They do not describe whether the calibration process happened to protect the specific capabilities a given deployment actually needs.
Leave a Reply