EAGLE-3 Speculative Decoding in Production
Speculative decoding pairs a fast draft model with the target model. The draft proposes K tokens; the target verifies in a single forward; accepted tokens are free. In 2026, EAGLE-3 is the production-grade variant — it trains a draft head on the target model's hidden states rather than on raw tokens, pushing acceptance rate alpha into the 0.6-0.8 band on general chat. The right question is not "how fast is the draft" but "what is alpha on my traffic?" If alpha drops below 0.55, speculative decoding is net negative at high concurrency because every rejected draft costs a second target forward pass. This lesson teaches you to measure alpha first and flip the flag second. Name the three generations of speculative decoding and explain what EAGLE-3 changes from EAGLE-2 and from a classic draft model. Define acceptance rate alpha, compute expected speedup from alpha and K (draft length), and identify the break-even alpha for your target concurrency. Explain why speculative decoding is opt-in (not default) in vLLM 2026 and why turning it on without measuring alpha is a production anti-pattern. Write a measurement plan: which benchmark, which prompt distribution, which concurrency point, which metric to gate on. Decode is memory-bound. On an H100 running Llama 3.3 70B FP8, each decoded token reads 140 GB/s of weights and emits one token.…
EAGLE-3 Speculative Decoding in Production: Speculative decoding pairs a fast draft model with the target model. The draft proposes K tokens; the target…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.