Cheat sheets

Speculative Decoding: Draft, EAGLE, MTP

Draft and verify with no quality loss. Vanilla, Medusa, EAGLE-3 and MTP drafts, speedup math, vLLM config, gates.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Draft types side by side#

FactorVanilla draftMedusaEAGLE-3MTP (DeepSeek-V3)
DraftSeparate small LLMExtra LM heads on verifierMulti-layer feature fusion + TTT1 transformer block + projection
Trained whenReuse an existing modelSFT stage on the verifierPost-pre-trainingDuring pre-training
Training costNone~1B SFT tokens~60-200B tokensAbout 10% slower pre-training
α0.55-0.700.65-0.750.88-0.920.80+ at depth 1
Speedup1.8-2.3×2-3×3.5-6.5×1.8×
Second modelYesNoTiny draft headNo, part of the model
Existing weightsYesNeeds fine-tuneYesNo, re-train

From Lesson 10.15, Lesson 10.18, Lesson 07.16

The loop in code#

for i, d in enumerate(drafts):
u = rng.random()
ratio = q_prob / p_prob if p_prob > 0 else float("inf")q verifier, p draft
if u < min(1.0, ratio):
emitted.append(d)
kv.extend(1)accepted: cache grows by one
else:
correction = sample(residual(q, p), rng)(q - p)+ renormalized
emitted.append(correction)
kv.truncate_to(prefix_len + len(emitted))KV rollback
return emitted, 1one verifier forward
bonus = sample(q, rng)all N accepted: free token
emitted.append(bonus)

From 10.15 code/main.py

Commands#

CommandWhat it does
vllm serve <model> --speculative-config '{"method": "eagle", "model": "<draft>", "num_speculative_tokens": 5}'EAGLE draft head
--speculative-config '{"method": "draft_model", "model": "meta-llama/Llama-3.2-1B-Instruct", "num_speculative_tokens": 5}'Vanilla draft model
--speculative-config '{"model": "<draft>", "num_speculative_tokens": 5, "method": "eagle3"}'EAGLE-3 draft
python main.py --vocab 64 --alpha 0.75 --k 4 --samples 50000Rejection rule, alpha sweep, tree mask demo

From Lesson 07.16, Lesson 10.15, 10.25 code/main.py

Troubleshooting#

SymptomCauseFix
No speedup after deploySpec decode is opt-inSet speculative_config explicitly
Mean ITL drops, P99 ITL risesRejected drafts take two passesWatch P99 ITL, tune before ship
α below 0.55 on trafficDraft trained on general dataTrain a domain draft or disable
α collapsesHigh-temperature creative samplingFall back to plain decode
No gain at high concurrencyBatching already fills the FLOPsMeasure break-even at that concurrency
Cache wrong after a rejectEntries past j-1 stay writtenTruncate to the accepted prefix
Gain is small on a tiny modelDraft not much cheaperSkip spec decode under 3B

From Lesson 17.05, Lesson 07.16, Lesson 10.15, Lesson 10.25

One speculative step#

1Draftthe cheap model $M_p$ proposes $N$ tokens withprobabilities $p_1 \ldots p_N$.2Verifythe verifier $M_q$ runs once on prefix plusdrafts. It returns $q_1 \ldots q_{N+1}$.3Acceptwalk left to right. Accept with probability$\min(1, q_i(d_i) / p_i(d_i))$.4Rejectat the first rejection, sample from $(q_j -p_j)_+$ normalized. Discard later drafts.5Bonuswhen all $N$ accept, sample one more token from$q_{N+1}$.6Roll backset the KV cache to the accepted prefix length.7Resultoutput is identically distributed to theverifier alone.

From Lesson 07.16, Lesson 10.15

Serving stack#

k tokensaccept, resampleclientRequestingresschat completionsprocessvLLM orSGLangone replica perGPUmodelDraftEAGLE-3 heads orngramgpuTargetFP8 or INT4quantizedprocessVerify passk draft tokensin one passclientToken streamaccepted prefixto clientstorePrometheusmetricsacceptance,queue wait, p99queueHPAscales on queuewaitk tokensaccept, resampleclientRequest ingresschat completionsprocessvLLM or SGLangone replica per GPUmodelDraftEAGLE-3 heads or ngramgpuTargetFP8 or INT4 quantizedprocessVerify passk draft tokens in onepassclientToken streamaccepted prefix toclientstorePrometheus metricsacceptance, queue wait,p99queueHPAscales on queue wait

From Lesson 19.14

Functions in the lessons#

TaskCode
Residual on rejectionresidual(q, p)
One step with KV rollbackspec_step(q, p, N, kv, rng)
Tokens per verifyexpected_tokens_per_verify(alpha, N)
Wall time per tokenwall_time_per_token(alpha, N, c)
Speedup for a configexpected_speedup(p)
Tree verify masktree_attention_mask(tree)
Distribution checkchi_square(observed, expected)
Measured acceptancem.acceptance_rate(k)

From 07.16 code/main.py, 10.15 code/main.py, 10.18 code/main.py, 10.25 code/main.py, 17.05 code/main.py, 19.14 code/main.py

Speedup math#

Tokens per verify
\mathbb{E}[\text{accepted}] = \frac{1 - \alpha^{N+1}}{1 - \alpha}
Example
α = 0.8, K = 4 gives 3.36 tokens per forward.
Wall time per token
(N * c + 1) / E[accepted], with c = cost(draft) / cost(verifier).
Biggest lever
acceptance rate α moves the speedup more than draft length N.
Production form
S(alpha, K) = (1 + K*alpha) / (1 + verify_overhead)
Break-even
alpha_breakeven = verify_overhead / K. It rises with concurrency.
MTP loss
L_{\text{MTP}} = \frac{\lambda}{D} \sum_{k=1}^{D} L_k, lambda 0.3 then 0.1.

From Lesson 10.15, Lesson 10.25, Lesson 17.05, Lesson 10.18

Script defaults#

SettingDefaultNote
--vocab32Vocab size, at least 2
--alpha0.75Target acceptance rate in (0, 1]
--k4Draft length, at least 1
--samples20000Sample count
--seed0Random seed
n_tokens1000simulate_tail tokens per run

From 10.25 code/main.py, 17.05 code/main.py

Use each draft when#

  • Use a vanilla draft for a fast prototype with no training.
  • Use Medusa heads when you can fine-tune the verifier.
  • Use EAGLE-2 or EAGLE-3 in production for maximum speed.
  • Use lookahead when you want no draft, no training and no extra params. Expect 1.3 to 1.6×.
  • Use MTP when you control pre-training and hidden size is at least 4096.
  • Use spec decode for chat latency, code and JSON output, and long generations.
  • Skip it for outputs under 50 tokens and batch-1 offline jobs.

From Lesson 07.16, Lesson 10.18, Lesson 17.05, Lesson 10.15

Abbreviations#

α
acceptance rate. Per-token chance that the verifier accepts a draft.
K or N
draft length. Typical 4-8 tokens per target forward.
EAGLE
tiny draft conditioned on the verifier's hidden states.
TTT
training-time test. The draft sees its own outputs during training.
MTP
multi-token prediction. A module predicts a token k positions ahead.
P99 ITL
the tail metric to watch after you enable spec decode. Not P50.
HPA
autoscales the server on queue wait, not CPU.

From Lesson 10.15, Lesson 17.05, Lesson 10.18, Lesson 19.14

Questions#

Does spec decode change output quality?
No. The output matches the verifier distribution exactly.
Why does decode leave room for this?
Decode is memory-bound. Compute sits idle.
How do I read alpha in vLLM V1?
Divide spec_decode_metrics.accepted_tokens_per_request by draft length.
What did EAGLE-3 change?
It drops the feature loss and trains with training-time test.
Why is MTP a good draft?
Sequential modules keep the causal chain, like a decoder.

From Lesson 07.16, Lesson 10.15, Lesson 10.18, Lesson 17.05