Quantization for Inference

Definition

Inference quantization stores/computes weights, activations, and/or the KV cache in fewer bits to buy HBM bandwidth and capacity. Because decode is bottlenecked on moving bytes, fewer bits directly speeds it up — but which scheme helps depends entirely on whether you are memory- or compute-bound (roofline). Three flavors, matched to three bottlenecks.

Key math / How it works

1. Weight-only (W8/W4, activations stay FP16).

• GPTQ (arXiv:2210.17323): second-order, one-shot post-training quantization.
• AWQ (arXiv:2306.00978): activation-aware — protects the salient weight channels.
• 4-bit weights ≈ 4× smaller weights, minimal accuracy loss. The matmul dequantizes on the fly. Ideal for memory-bound, small-batch/latency decode. Adds no tensor-core speedup — the math is still FP16.

2. Weight + activation (W8A8 / FP8).

• SmoothQuant (arXiv:2211.10438): migrates activation outliers into the weights so both quantize to INT8.
• FP8 (E4M3, arXiv:2209.05433): hardware-accelerated on Hopper/Blackwell tensor cores → compute and bandwidth wins. Increasingly the default for high-throughput serving.

3. KV-cache quantization (KV8/KV4). Quantize the cache itself to fit more/longer sequences — attacks the KV-cache memory wall directly; some accuracy risk on long context.

Why it matters

Match the scheme to the bottleneck. Weight-only 4-bit helps latency-bound small-batch decode (bandwidth is the wall) but does little for compute-bound prefill or large-batch serving — there the on-the-fly dequant can even hurt, and you want W8A8/FP8 that speeds the matmul itself. A 4-bit weight-only kernel that shines at batch 1 gives little at batch 64, because rising intensity pushes decode toward compute-bound. “Which precision” is a roofline question — diagnose the regime first. Contrast with training-time mixed-precision-training, which targets throughput/stability rather than serving bandwidth.

Taught in

• 12-llm-inference-systems — §7 the three flavors and the match-to-bottleneck rule.
• 13-gpu-engineer-roadmap — §4 numerics/precision as a first-class skill.

See also

• prefill-decode — which phase each scheme helps
• kv-cache — target of KV-quant
• roofline-model · arithmetic-intensity · mixed-precision-training
• gpu-systems-for-llms