Speculation won the benchmark and lost in production

The L4 sweep

The default server against the EAGLE-3 server with the Tengyunw/qwen3_8b_eagle3 draft, three seeds per point, 512 tokens in and 128 out:

The acceptance length was 2.73 to 2.74 tokens per step at every concurrency. Each token came out 1.6 to 2.3 times faster at every load. From concurrency 32 the server delivered no more tokens per second than the default.

The pool explains the first half

The startup log of the EAGLE-3 server shows the price of a draft on a small card. The target alone had a pool of 24,094 tokens. With the draft loaded, the pool held 9,951 tokens, about 15 requests of 640 tokens, and the draft took nearly 60 percent of the pool. With 15 requests in the pool, the rest waited for a slot. At concurrency 32 the median first token was 8,040 ms with EAGLE-3 and 2,970 ms without it.

The same log line shows max_running_requests=48. The speculative handlers set that cap when you do not set it, with one warning that scrolls away before the first request. On the GB10, where the pool had room, the cap was the limit instead: at 64 users throughput was still higher with speculation, 276.32 against 235.38 tokens per second, but the median first token was 7,772 ms against 557 ms, because 16 requests waited behind the 48.

The dataset explains the second half

The random dataset at this version takes the first turn of ShareGPT conversations and cuts or repeats it to 512 tokens, with the end token ignored. The manual then ran the same L4 sweep on four datasets with the clock locked at 900 MHz and a flush before each run:

EAGLE-3 cut the time per token about in half at 8 users on every dataset, with an acceptance length of 2.35 to 2.78. At 32 users it gave 1.6 times the output on GSM8K and code. On the random set and ShareGPT it gave no more output, and its first token came 2 to 3 times later, because the draft took pool space that the queue needed. NGRAM helped only on GSM8K and code, by about 1.1 to 1.2 times at 8 users, and on the other two sets it was slower, with a P99 ITL of 244 to 779 ms.

The card and the model decide

The same feature on the GB10 with the mixture-of-experts Qwen3-30B-A3B held its gain at every load, 1.56 to 1.89 times shorter time per token, and still gave 1.17 times the throughput at 64 users. The manual's reading: a dense model reads all its weights per step, so a larger batch already uses the spare compute, while the expert model reads only the experts that the step's tokens chose and the verify pass scores four positions with weights it already read. That is one model and one GPU.

Other teams found the same limits. A benchmark study found that synthetic prompts overstated speculative throughput by 23 percent on average, that the best draft length fell to 1 as the batch grew, and that random token ids left 20 to 30 percent of the experts idle in some layers. A 2025 study of serving benchmarks saw the median TPOT 1.3 times lower and the P99 TPOT 1.16 times higher with speculation, and a later first token because the draft runs its own prefill.

What to do

1. Read the KV Cache is allocated line of the speculative server and compare it with the plain server.
2. Set --max-running-requests yourself, from the concurrency where your sweep shows speculation still wins.
3. Benchmark with your own prompts and real output lengths, with --disable-ignore-eos, and read accept len on them.
4. Compare the P99 ITL and the P99 TTFT, not only the median.
5. Turn speculation on for interactive traffic with a few users per replica; leave it off for batch jobs and for high concurrency on a card where the pool is the limit.