Cheat sheets

Inference Metrics and Load Testing

Find why a load test lies. TTFT, TPOT, goodput, P99, realistic prompts, retry storms and hedged reads.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Symptoms, causes, fixes#

SymptomCauseFix
Test held at 500 users, prod fell at 200Identical prompts hit the prefix cache--mean-input-tokens plus --stddev-input-tokens
Locust ITL high, server is fineGIL trap: client tokenization backlogLLM-Locust or a compiled harness
GenAI-Perf TPOT 6 ms, LLMPerf 11 msGenAI-Perf excludes TTFT from ITLState the tool and the definition
High throughput, users leaveLow goodput, tail blows the SLOGate deploys on goodput
Mean TPOT 9 ms, users complainRight-skewed tail, P99 TPOT 65 msReport P50, P90, P99
TPOT spikes to 50 msLong prefill on a neighbor sequenceChunked prefill
All clients retry at the same timeSynchronized exponential waits--jitter=true and a shared --budget
Hedge looks freeCancel does not stop server workCount fixture work next to latency

Metric formulas#

TTFT = queue_time + network_request + prefill_timebefore anything streams
TPOT = (decode_forward_time + scheduler_overhead) / tokens_producedsame as ITL
E2E = TTFT + TPOT * output_tokens + network_responseTPOT-dominated above 500 tokens
throughput = total_output_tokens / elapsed_timefleet efficiency only
goodput = fraction of requests meeting (TTFT <= a) AND (TPOT <= b) AND (E2E <= c)

SLO profiles in main.py#

Factorloosetargettight
TTFT ms800500300
TPOT ms251510
E2E ms300020001500
VerdictSHIPPABLE at 99%DEGRADED at 95%FAILING below

Four load patterns#

FactorSteady-stateRampSpikeSoak
ShapeConstant RPS0 to target RPSSudden 3-10x RPS, then backSteady-state
Duration30-60 min15 min2 min4-8 hours
CatchesBaseline regressionsCapacity breakpoint, warm-up anomaliesAutoscaling latency, queue saturation, cold startMemory leaks, connection-pool drift

Use each tool when#

LLMPerf
a benchmark run. Rust-backed tokenization, mean/stddev prompts.
GenAI-Perf
you need the NVIDIA reference. ITL excludes TTFT.
LLM-Locust
your team uses Locust. It fixes the GIL trap.
guidellm
large-scale synthetic benchmarking.
k6 v2026.1.0
CI/CD gates and SLA tests. Streaming-aware, no GIL.
k6 Operator
distributed tests through TestRun / PrivateLoadZone CRDs.
Vegeta
constant-rate saturation of a gateway or rate limit.
Locust 2.43.3 stock
never alone for LLMs. It has the GIL trap.

Lab error strings#

CommandWhat it does
invalid bounded policyBase below 1, cap below base, or attempts over 10
non-retryable HTTP statusStatus is not 429, 502, 503 or 504
server delay exceeds policy capRetry-After is above the policy cap
retry attempts or shared budget exhaustedNo attempts or budget left
winner requires HTTP 200A fast error cannot win the hedge
body exceeds 64 KiBResponse body is over the read limit
no valid responseBoth hedge attempts failed
samples must be 1..100Sample count is out of range

Run the labs#

python3 scripts/project_test.py retry-storm-lab --init /tmp/retry-storm-lab-work
python3 scripts/project_test.py retry-storm-lab --all --solution --strict
cd projects/retry-storm-lab/solution
go run . --clients 20 --budget 60 --jitter=false --out /tmp/synchronized.jsonsynchronized waits
go run . --clients 20 --budget 60 --jitter=true --seed 7 --out /tmp/jittered.jsonseeded spread
go run . --local-http --clients 20 --budget 12 --out /tmp/retry-wire.jsonreal loopback HTTP
node --experimental-strip-types cli.ts /tmp/retry-wire.jsonverify counters
cd projects/hedged-request-lab/solution
go run . --samples 3 --delay-ms 10 --out /tmp/hedge-report.json80ms primary, 5ms secondary

Lab flag defaults#

SettingDefaultNote
--clients20Concurrent clients
--budget12Shared retry attempts
--base-ms10Initial backoff
--max-ms80Maximum retry wait
--attempts4Maximum attempts per client
--capacity5Simulated capacity per exact millisecond
--delay-ms10Hedge delay
--samples3Samples per policy

One hedged read#

1Clientstarts the primary GET and a delay timer.2Clienta valid primary before the timer prevents thehedge.3Clientelse launches at most one secondary. Delay is 0to 1 second.4Validatoronly HTTP 200 with a nonempty JSON value canwin.5Clientcancels the loser and observes it for at most100ms.6Receiptrecords late and unobserved attempts and winnerlatency.7Fixturecounts server work that continues after cancel.

Worked numbers#

TTFT 500 ms, 100 tokens in 700 ms decode?
GenAI-Perf ITL = 700/99 = 7.07 ms. LLMPerf ITL = 1200/100 = 12.00 ms.
Twenty clients, four attempts, budget 12?
At most 32 total calls.
Base 10, cap 25, no jitter?
Waits of 10,20,25,25.
Three baseline and three hedged samples?
Three calls versus six. The fixture records nine.
Samples [2,4,10,20,80]?
P50=10, P95=80, maximum=80.
Primary HTTP 500 at 1ms, secondary valid at 15ms?
The secondary wins. A fast error cannot win.

Reference numbers and gates#

  • Llama-3.1-8B-Instruct on TRT-LLM: mean TTFT 162 ms, mean TPOT 7.33 ms.
  • Consumer 70B SLO: TTFT P99 <= 800 ms, TPOT P99 <= 25 ms, goodput >= 99%.
  • Run at 2x peak concurrency. Take 30-50 iterations.
  • Use --mean-input-tokens 800 --stddev-input-tokens 300 --mean-output-tokens 150.
  • CI gate: P50/P95 TTFT, 5xx < 5%, TPOT under threshold. Break the build on breach.
  • Publish tool name, tool version, model, hardware, concurrency and prompt distribution.