Cheat sheets

Inference Optimization: KV Cache, Quantization, Batching

Size the KV cache, batch continuously, pick a quantization format, and measure what users feel.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Tasks#

TaskCode
Reuse KV of shared prompts--enable-prefix-caching
Store the KV cache in FP8--kv-cache-dtype fp8
Split the model over 4 GPUs--tensor-parallel-size 4
Cap the sequence length--max-model-len 32768
Set the HBM share for KV blocks--gpu-memory-utilization 0.9
Turn on EAGLE-3 draftsspeculative_config with "method": "eagle3"
Read the acceptance per requestspec_decode_metrics.accepted_tokens_per_request
Use Flash Attention in transformersattn_implementation="flash_attention_2"

From Lesson 07.12, Lesson 17.04, Lesson 17.05, Lesson 17.08

Serve a model#

pip install vllminstall the engine
vllm serve meta-llama/Llama-3.1-70B-Instruct \model to serve
--tensor-parallel-size 4 \weights split over 4 GPUs
--max-model-len 32768 \longest sequence
--enable-prefix-caching \reuse KV of shared prefixes
--kv-cache-dtype fp8FP8 KV, test accuracy

From Lesson 07.12, Lesson 17.09

Offline batch in Python#

from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Meta-Llama-3-70B-Instruct",
tensor_parallel_size=4,4 GPUs
enable_prefix_caching=True,skip shared prefill
max_model_len=8192,context cap
gpu_memory_utilization=0.9,HBM share for KV blocks
)
params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(["<prompt>"], params)continuous batching inside

From Lesson 10.12, Lesson 17.04

Formulas#

KV cache size
2 * num_layers * num_kv_heads * head_dim * seq_len * bytes_per_param.
Llama 3 70B, BF16
320 KB per token, 1.28 GB at 4,096 tokens, 40 GB at 128K.
ops:byte
FLOPs per token over bytes read. Decode at batch 1 is ~1, so memory-bound.
TTFT
queue_time + network_request + prefill_time.
E2E
TTFT + TPOT * output_tokens + network_response.
Spec speedup
(1 + K * alpha) / (1 + epsilon). K=5, alpha=0.7 gives 4.1x.
INT8 scale
max(abs(tensor)) / 127, then clamp(round(tensor / scale), -128, 127).
Goodput
share of requests that meet every TTFT, TPOT and E2E bound.

From Lesson 10.12, Lesson 10.11, Lesson 17.05, Lesson 17.08

Serving engine#

admitbatchallocatemapsforwardverifyWaiting queuenew requestsSchedulerruns every decode stepRUNNING listprefill and decodeKV block pool16-token blocksBlock tablelogical to physicalFused forwardone GPU callDraft headproposes K tokensadmitbatchallocatemapsforwardverifyWaitingqueuenewrequestsSchedulerruns everydecodestepRUNNINGlistprefilland decodeKV blockpool16-tokenblocksBlocktablelogical tophysicalFusedforwardone GPUcallDraftheadproposes Ktokens

From Lesson 17.04, Lesson 07.12, Lesson 17.05

Defaults worth knowing#

SettingDefaultNote
PagedAttention block16 tokensFragmentation under 4 percent
Chunked prefill slice512 tokensProtects the P99 ITL
--gpu-memory-utilization0.9HBM for KV after weights
speculative_configopt-inNo config, no acceleration
GGUF quantQ4_K_MCPU and edge serving
Datacenter INT4AWQUnless you need multi-LoRA
PTQ calibration data128-1024 examplesMatch the deployment domain

From Lesson 17.04, Lesson 17.05, Lesson 17.09, Lesson 10.11

One scheduler iteration#

1Schedulerremoves sequences that hit EOS or max_tokens.2Allocatorreturns the blocks of finished sequences to thepool.3Scheduleradmits waiting sequences while free KV blocksexist.4Scheduleradds a prefill chunk of 512 tokens for promptsin prefill.5Scheduleradds one decode token for every other runningsequence.6GPUruns the whole batch in one fused forward.7Allocatoradds one block when a sequence grows past itsblocks.

From Lesson 17.04

Pick a quantization format#

CommandWhat it does
GGUF Q4_K_MCPU and edge, through llama.cpp
GPTQMulti-LoRA on vLLM, Marlin kernels
AWQGPU chat without LoRA. ~741 tok/s on 7B
FP8Reasoning, code, medical. Near-lossless
NVFP4Blackwell. Validate quality per model
FP8 KVSeparate choice. Affects attention accuracy

From Lesson 17.09, Lesson 10.11

Troubleshooting#

SymptomCauseFix
P99 ITL jumps on long promptsPrefill blocks decodeChunked prefill in 512-token slices
Spec decode slows the tailAlpha below 0.55Disable it or train a domain draft
Mean ITL falls, P99 ITL risesRejected drafts need two passesWatch P99 and tune
4-bit model still runs out of HBMKV cache stays FP16 or FP8Budget weights, KV, activations
Domain quality drops after INT4Calibrated on generic web textCalibrate on in-domain data
MATH score dropsAWQ INT4 loses 3 to 5 pointsUse FP8 or BF16
Tools report different TPOTOne tool excludes TTFT from ITLState the tool and definition
Two features fail togetherCombination not supportedCheck the compatibility matrix

From Lesson 17.04, Lesson 17.05, Lesson 17.08, Lesson 17.09

Glossary#

TTFT
time to first token. Queue plus network plus prefill.
TPOT
time per output token. Decode cost per token after the first.
ITL
inter-token latency. Same as TPOT in most tools.
E2E
end to end latency of one request.
P99
the 1 in 100 worst latency. The user experience metric.
HBM
high bandwidth memory. 80 GB on H100, ~3 TB/s.
Alpha
share of draft tokens that the target accepts.
PTQ
post-training quantization. No retraining.

From Lesson 17.08, Lesson 07.12, Lesson 17.05, Lesson 10.11

Q&A#

Why is decode memory-bound?
It does ~1 operation per weight loaded at batch 1.
Why does GQA matter?
Llama 3 70B with 8 KV heads uses 4 KB per token per layer.
Is speculative decoding exact?
Yes. The output distribution equals the target model's.
Is Flash Attention approximate?
No. The running softmax keeps the result exact.
Why goodput over throughput?
High throughput at 60 percent goodput is failure.

From Lesson 10.12, Lesson 07.12, Lesson 17.08

Tips#

  • Report P50, P90 and P99. Never report only the mean.
  • Measure alpha on real traffic before you turn on spec decode.
  • Use prefix caching for agents with repeated tool prompts.
  • Calibrate on in-domain data. Hundreds of samples is usually enough.
  • Benchmark at 2x peak concurrency for 30 to 50 iterations.
  • Publish tool, version, model, hardware, concurrency and prompt mix.

From Lesson 17.08, Lesson 17.05, Lesson 07.12, Lesson 17.09