Infrastructure & serving · Glossary term
What is Dynamic Batching?
A runtime policy that forms inference batches from queued requests according to compatible shapes, maximum size, priority, and allowed queue delay.
Why does Dynamic Batching matter?
Grouping requests can improve hardware utilization, but waiting for a batch can make latency worse when traffic is sparse or requests differ sharply.
Dynamic Batching in practice
Set queue-delay and batch limits from measured latency objectives, separate incompatible request shapes, and compare throughput with tail latency at realistic arrival rates.
What is the common confusion about Dynamic Batching?
Dynamic batching assembles batches from queued work. Continuous batching changes membership while autoregressive generation is already running.
Learn Dynamic Batching in the course
Start with
- Serving Engine Internals — PagedAttention, Continuous Batching, Chunked Prefill
Modern serving-engine throughput rests on three compounding defaults, not a single trick. PagedAttention is always on. Continuous batching injects new requests into the active batch between decode…
Taught in Phase 17: Infrastructure & Production.
Related terms
- Admission ControlA pre-acceptance gate that decides whether a request may enter a bounded queue or service under the system's current capacity, priority,…
- Continuous BatchingA serving scheduler that adds and removes generation requests at iteration boundaries instead of waiting for every request in a fixed…
- SaturationThe degree to which a constrained resource or service has exhausted its capacity, including queued work that cannot begin promptly.
- Tail LatencyThe latency experienced by the slowest portion of requests, commonly summarized with a high percentile under a stated workload and time…
- Chunked PrefillA serving technique that divides a long prompt's prefill work into smaller schedulable pieces so prompt processing can interleave with…
Sources
More terms in Infrastructure & serving
This entry comes from glossary/terms.md on GitHub. Browse all 250 glossary terms.