Manuals / vLLM 101 / Part 8 · How vLLM Works
8.7

How speculative decoding checks drafts

This page explains draft proposal, verification, and acceptance.

Speculative decoding promises more tokens for each step without a change to the output. When you finish this section, you can explain how vLLM keeps that promise, and why the gain shrinks as the batch grows.

Guess, then check

A decode step on a small batch reads all the weights to produce one token for each request. Most of the GPU sits idle while it waits for memory. Speculative decoding uses that idle time.

A cheap proposer guesses the next few tokens. The target model then runs once over the guesses and computes its own distribution at every position. A rejection sampler walks the guesses in order. It keeps each guess that the target model's distribution supports, and at the first rejection it samples a correct token from the target and stops. One step therefore gives at least one token and at most the number of guesses plus one.

The check is what keeps the output the same. With greedy sampling, a guess stays only if it equals the target's top token. With random sampling, the rejection rule keeps the output distribution equal to the target's distribution.

Fig. 8.4one verify passsequence
One verify pass keeps the drafts up to the first rejection, plus one token Top: a proposer drafts three tokens, the target model checks them in one pass, keeps the first two, rejects the third, and adds its own token. Bottom: one real request with Qwen3-8B on our L4 and EAGLE3 with three drafts per step wrote 12 tokens. One came from prefill and 11 from 6 verify passes, which kept 5 of 18 drafts. THE RULE, BY POSITION draft 1 draft 2 draft 3 kept kept rejected target token One target pass writes the kept drafts and one token of its own. ONE REAL REQUEST: QWEN3-8B ON OUR L4, EAGLE3, 3 DRAFTS Paris 12095 . 13 The 576 capital 6722 of 315 Italy 15344 is 374 Rome 21718 . 13 The 576 capital 6722 of 315 Token 1 came from prefill. The other 11 came from 6 verify passes. Drafts kept per pass: 0 in 3 passes, 1 in 2 passes, 3 in 1 pass. So 5 of 18 drafts were kept, about 1.8 tokens per pass.
One target pass keeps the drafts up to the first rejection and adds one token, so one real request got about 1.8 tokens per pass. The top half shows the rule by position. The bottom half is a real Qwen3-8B request on our L4, served by SGLang with EAGLE3, the same rule vLLM uses. Green tokens came from verify passes.

The proposers

vLLM 0.31.0 has several kinds of proposer, set with --speculative-config:

MethodWhere the guesses come from
ngramearlier text in the same prompt that matches the last few tokens
suffixa suffix tree over earlier prompts and answers
eagle, eagle3a small draft head that reads the target model's hidden states
*_mtpextra prediction heads that come with some models, such as DeepSeek
draft_modela separate small model

N-gram and suffix methods cost almost nothing, and they help when the answer copies the prompt, as in code edits. A draft head like EAGLE3 needs a trained head for your model, and it helps on open text. For Qwen/Qwen3-8B, a public EAGLE3 head exists with one layer and a draft vocabulary of 32,000 tokens.

Why the gain falls with load

The idle time that speculation uses exists only while the step reads more memory than it computes. As the batch grows, the step does more compute for each byte, and the idle time goes away. The checks of rejected guesses then cost real compute. The vLLM documentation rates the gain of EAGLE as high at a low request rate and lower at a high one.

The dynamic mode follows from this. It sets the number of guesses from the batch size and stops guessing for large batches. At 0.31.0, its online example runs on the older model runner.

What to watch

The engine exports the proposed and the accepted tokens. The acceptance rate and the mean number of tokens for each step tell you whether the proposer guesses well on your text. A high acceptance rate with no gain in the time per token means the batch is already compute-bound.

vLLM 0.31.0 · db9527a · 2026-10-05 · Edition 2026.10