Cheat sheets

LLM Serving with SGLang

Start, query, size, measure and fix an SGLang server. Pinned to SGLang 0.5.22.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Tasks#

TaskCode
Start a serversglang serve --model-path <model> --port 30000
Native requestPOST /generate {"text": "<prompt>"}
OpenAI-style requestPOST /v1/chat/completions
Readiness probeGET /ready
Read metricsGET /metrics
Benchmark at a fixed concurrency cap--max-concurrency 8
Benchmark at a fixed request rate--request-rate <rps>
Stop with a draindocker stop <container>

From sglang-101 capture: server-default.command.txt, sglang-101 capture: one_request.py, sglang-101 capture: run.py, sglang-101 §5.1, sglang-101 §5.2, sglang-101 §7.1

Start the server (terminal 1)#

sglang serve --model-path <model> \weights to load
--port 30000 \HTTP port
--enable-cache-report \cached_tokens in usage
--enable-metrics \turns on /metrics
--mem-fraction-static 0.4weights plus KV pool share

From sglang-101 capture: server-default.command.txt, sglang-101 §R1, sglang-101 §2.1, sglang-101 §3.2

Benchmark it (terminal 2)#

Concurrency cap of 8. The server in terminal 1 must be up.
python -m sglang.benchmark.serving --backend sglang \
--port 30000 --dataset-name random \
--random-input-len 512 --random-output-len 128 \
--random-range-ratio 1.0 \default 0.0 halves prompts
--num-prompts 32 --max-concurrency 8255.63 tok/s on the capture

From sglang-101 capture: run.py, sglang-101 §5.1, sglang-101 §R1, sglang-101 capture: bench-random-c8.txt, sglang-101 capture: bench-random-c32.txt

Key flags#

CommandWhat it does
--mem-fraction-static <f>Memory share for weights plus KV pool
--max-total-tokens <n>Fixes the KV pool size in tokens
--max-running-requests <n>Caps decoding requests. Set it at the knee
--chunked-prefill-size <n>Largest prompt piece in one step
--schedule-policy lpmLongest prefix match order
--schedule-conservativeness 1.3Higher forecast, fewer retractions
--disable-radix-cacheNo prefix reuse. Only to measure
--tp-size <n>Splits each layer over n GPUs

From sglang-101 §R1, sglang-101 §2.2, sglang-101 §2.3, sglang-101 §3.2, sglang-101 §6.1, sglang-101 §5.1

Three processes#

HTTPZMQZMQZMQmatchslotsclientClientHTTP :30000processTokenizerManagermain process, tokenizegpuSchedulerowns the GPUprocessDetokenizerManagerids to textstoreUnifiedRadixCacheprefix treestoreKV poolone slot per tokenHTTPZMQZMQZMQmatchslotsclientClientHTTP :30000processTokenizerManagermainprocess,tokenizegpuSchedulerowns theGPUprocessDetokenizerManagerids to textstoreUnifiedRadixCacheprefix treestoreKV poolone slotper token

From sglang-101 §2.1, sglang-101 §3.1, sglang-101 §3.3

Key metrics#

sglang:time_to_first_token_seconds
histogram. Alert on p99 over 5 min.
sglang:inter_token_latency_seconds
histogram. Shows stream stutter.
sglang:queue_time_seconds
histogram. Wait before first prefill.
sglang:num_queue_reqs
gauge. Above 0 for minutes means past the knee.
sglang:token_usage
gauge. Near 1.0 means the pool is full.
sglang:num_retracted_reqs
gauge. Alert above 0.
sglang:prefill_effective_tokens_total
counter. Windowed hit rate from its modes.
sglang:cache_hit_rate
gauge. Last log interval only.

From sglang-101 §R2, sglang-101 §5.2

Log lines to read#

CommandWhat it does
bytes_per_slot=114688, pool_size=181574KV bytes per token, pool slots
max_total_num_tokens=181574Pool size SGLang computed
max_running_requests=4096Computed request cap
#new-token: 3, #cached-token: 2New plus cached is the prompt
#running-req: 5, ... #queue-req: 11Admission holds requests back
KV cache pool is full. Retract requests.Retraction. A performance alarm
crashed with exit code -15Our SIGTERM, not a fault
Insufficient MemoryOut of memory outside the pool

From sglang-101 capture: server-default.log, sglang-101 capture: server-small-pool.log, sglang-101 capture: oom-exploratory.log, sglang-101 §2.1, sglang-101 §2.3

One request, end to end#

1TokenizerManagerapplies the chat template, tokenizes, sends itover ZMQ.2Scheduleradds it to the waiting queue in policy order.3RadixCachematch_prefix finds the cached prefix, shown as#cached-token.4Scheduleradmits it if prompt plus max_new_tokens fits thepool.5Schedulerprefills #new-token in chunks and samples thefirst token.6Schedulerdecodes one token per step until a stopcondition.7DetokenizerManagerturns ids into text for the TokenizerManager tostream.

From sglang-101 §2.1, sglang-101 §2.2, sglang-101 §2.3, sglang-101 §3.3

Defaults in the code#

SettingDefaultNote
--mem-fraction-staticNoneDerived at startup. Kit sets 0.4
--max-running-requestsNoneDerived from pool and context
--chunked-prefill-sizeNoneDerived from GPU memory
--schedule-conservativeness1.0Raise it on retractions
--schedule-policyfcfsTry lpm for long shared prefixes
--enable-cache-reportFalseSet it always
--enable-metricsNoneResolves to off
--watchdog-timeout300 sLong forward passes

From sglang-101 §R1

Troubleshooting#

SymptomCauseFix
Metal Insufficient MemoryPool took 87 percent of free memory--mem-fraction-static 0.4
RuntimeError on first batch, MLX--disable-overlap-scheduleKeep overlap on
Retractions again and againAdmission forecast too boldRaise --schedule-conservativeness
High TTFT, fast TPOTPast the knee, requests queueCap concurrency at the knee
No cached_tokens in usageNo --enable-cache-reportAdd the flag
#cached-token near 0Changing id at prompt startPut shared text first
"health check failed" under loadNo detokenizer output for 20 sUse /ready for readiness
Crash line at shutdownSIGTERM to the process groupSIGTERM to the main process

From sglang-101 §R3, sglang-101 §2.3, sglang-101 §1.2, sglang-101 §3.4, sglang-101 §5.1

Glossary#

TTFT
time to first token. Send to first streamed chunk.
TPOT
time per output token, after the first token.
ITL
inter-token latency. Gap between two streamed chunks.
KV cache
stored keys and values of earlier tokens.
TP
tensor parallelism. Layer weights cut into slices per GPU.
DP
data parallelism. Full model copies with own KV cache.
EP
expert parallelism. MoE experts divided among GPUs.
PD
prefill-decode disaggregation. Separate prefill and decode servers.

From sglang-101 §R4, sglang-101 §3.3, sglang-101 capture: bench-shared-prefix.txt

Q&A#

Why three processes?
The GIL. Text work must not delay GPU steps.
How many 256 in, 512 out requests fit 4,096 tokens?
Five. Each reserves 768.
Why 2,998 cached of 2,999?
The last prompt token always prefills.
Gauge 0.942, counters 0.636?
Gauge is the last interval. Trust the counters.
What is the knee?
The cap where throughput stops rising. Here at or below 8.

From sglang-101 §2.1, sglang-101 §2.3, sglang-101 §3.3, sglang-101 §5.2, sglang-101 §5.1

Tips#

  • Pass --random-range-ratio 1.0 in every benchmark.
  • Report throughput with p50 and p99 TTFT and TPOT.
  • Raise --mem-fraction-static in steps of 0.02 only.
  • Keep the radix cache on. Measure hit rate per feature.
  • Pin lmsysorg/sglang:v0.5.22, never latest.
  • Mount ~/.cache/sglang on a persistent volume.

From sglang-101 §5.1, sglang-101 §3.2, sglang-101 §3.3, sglang-101 §7.1