Continuous Batching

Definition

Continuous batching (a.k.a. iteration-level scheduling; introduced by Orca, Yu et al., OSDI 2022) schedules the serving batch at the granularity of a single decode step rather than a whole request. After every generated token the scheduler can evict finished sequences and admit waiting ones, keeping the GPU full of active sequences at all times.

Key math / How it works

The problem with static batching. Group $B$ requests, run them together, and you cannot return the batch until the longest sequence finishes. Because response lengths vary wildly, short replies sit idle behind long ones and new arrivals wait for the whole batch to drain — GPU utilization craters.

Iteration-level scheduling. Treat one decode step as the scheduling unit. Each step the scheduler rebuilds the running batch: drop sequences that emitted EOS, pull queued requests in. The GPU always runs a full batch of live tokens; effective batch stays high despite length skew.

Why it lifts throughput. Decode is memory-bound; its arithmetic intensity $\sim b$. Keeping $b$ maximal at every step (rather than letting it decay as sequences finish) amortizes the weight reads over more sequences → higher intensity → more tokens/s (roofline).

Package deal with paging. Constantly adding/removing sequences of unpredictable length only works if KV memory can be allocated and freed in small increments without fragmentation — exactly what PagedAttention provides. Together they convert “GPU idle behind the longest request” into “GPU always running a full batch.”

Why it matters

Continuous batching plus paged-attention is the backbone of modern high-throughput serving (vLLM, SGLang, TensorRT-LLM). It is the single scheduling change that most directly raises tokens/s under real, bursty, variable-length traffic — the axis serving is optimized for.

Taught in

• 12-llm-inference-systems — §5 static vs. iteration-level batching and the paging synergy.
• 13-gpu-engineer-roadmap — Project 5, continuous batching in a toy inference loop.

See also

• paged-attention — makes admit/evict memory-cheap
• kv-cache — what each admitted sequence holds
• prefill-decode — the memory-bound phase this batches
• arithmetic-intensity · roofline-model
• gpu-systems-for-llms