Operating SGLang

The first production incident with a new serving engine is rarely about tokens per second. A rolling update kills ten requests in the middle of their answers. A new pod takes four minutes to become ready, so the autoscaler starts three more. A liveness probe times out under load and restarts the busiest pod. None of these appear in a benchmark, and all of them cost more than a slow kernel.

When you finish this section, you can choose an image tag and keep startup short. You can connect the health probes to the right endpoints and stop a server without losing work. You can also change settings in the order that this manual measured, and read three production reports for what each one teaches.

Pin the image you run

The SGLang team publishes images on Docker Hub as lmsysorg/sglang. At v0.5.22 the latest tag holds the newest stable release, and latest-runtime holds a smaller image of the same release. Nightly tags follow main. The images contain CUDA 13 only. The docs name lmsysorg/sglang:v0.5.19-cu129 as the last CUDA 12 tag, so a host with an older driver needs that tag or a driver upgrade.

The documented start command gives the container a large shared memory segment and the host IPC namespace:

docker run -it --gpus all \
  --shm-size 32g \
  -p 30000:30000 \
  -v "$HF_CACHE_DIR:/root/.cache/huggingface" \
  --ipc=host \
  lmsysorg/sglang:latest bash

Our position: never deploy latest. SGLang tagged v0.5.20, v0.5.21, and v0.5.22 within four weeks, from 2026-09-18 to 2026-10-13, and each release changed defaults that this manual measures. A pinned tag such as v0.5.22 makes a regression a choice that you made. The tag latest makes it something that happened to you on a restart.

What pip install leaves out

The image exists for a reason that a pip install hides. We measured this on a host with two NVIDIA L4 GPUs, Ubuntu 24.04, a driver, and no CUDA toolkit. pip install sglang==0.5.22 finished with no error. The first sglang serve then failed, and so did the next three, each for a new reason:

=== attempt: out-l4-nopyheaders/server-default.log
<site-packages>/triton/backends/nvidia/driver.c:9:10: fatal error: Python.h: No such file or directory
…
=== attempt: out-l4-nonvcc/server-default.log
[2026-10-16 14:21:19] RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist
…
=== attempt: out-l4-jitmismatch/server-default.log
…
<site-packages>/flashinfer/data/cccl/libcudacxx/include/cuda/std/__cccl/cuda_toolkit.h:41:8: error: #error "CUDA compiler and CUDA toolkit headers are incompatible, please check your include paths"
…
=== attempt: out-l4-nolibcudart/server-default.log
/usr/bin/ld: cannot find -lcudart: No such file or directory

All four failures come from one fact. SGLang, Triton, and FlashInfer compile kernels on the target machine at startup, and a compiler needs a complete toolchain.

1. Triton compiles a small C helper against the Python headers. The system Python of Ubuntu has no Python.h. We changed to a Python build that includes its headers, from uv.
2. SGLang looks for nvcc under /usr/local/cuda. The pip wheels include nvcc 13.4 under nvidia/cu13 in site-packages, but nothing points to it. We set CUDA_HOME to that directory.
3. FlashInfer then compiled its attention kernels with that nvcc against its own bundled headers, and the versions did not match. We installed flashinfer-jit-cache, which holds kernels compiled in advance for SM 8.9.
4. The JIT kernels of SGLang link with -lcudart from $CUDA_HOME/lib64. The wheel has only lib/libcudart.so.13. We made a CUDA_HOME with a lib64 directory that holds libcudart.so.

The fifth start served requests. Our GB10 never had this problem, because its DGX operating system has a full toolkit in /usr/local/cuda.

The benchmark client has one more dependency that a firewall exposes. Its random dataset takes its text from ShareGPT and downloads that file from the Hugging Face Hub. With HF_HUB_OFFLINE=1, our first GPU benchmark failed although every model was on disk. We did not keep that log. Use --dataset-name random-ids on a machine without network access, or pass a local ShareGPT file with --dataset-path.

Our position: run SGLang from the pinned image or on a host with a full CUDA toolkit. A pip install on a host with only a driver works after our four repairs. Each SGLang upgrade can need them again.

Cold start is weights plus compilers

A new SGLang process loads the weights, compiles or loads kernels, sizes the KV pool, and captures graphs. Our capture logs each phase on one line. This run used a model that was already on disk:

[2026-10-16 13:58:06] Engine startup timings (s): load_weight=0.00, kv_cache_allocation=0.00, scheduler_e2e=7.01, cuda_graph={prefill=0.00, decode=0.00, target_verify=0.00, draft_prefill=0.00, draft_decode=0.00, draft_extend=0.00}, tokenizer_e2e=11.92

We measured 18 s from the first log line to "The server is fired up and ready to roll!" for a 0.6B model on the Mac. A large model on GPUs is a different problem. A field report from Doubleword, an engineer blog dated 2026-04, timed SGLang v0.5.10 with Qwen3.5-122B-A10B-FP8 on 8 B200 GPUs. A cold start took 695 s, and 531 s of that was weight loading. With kernel caches kept between runs, a warm start took 88 s. With a process checkpoint and weights staged in host memory, it took 9.6 s. Raw runs

The code helps with the cheap part of that work. When Python imports sglang, it points the Triton, TorchInductor, CUDA, TileLang, and FlashInfer caches into one directory, SGLANG_CACHE_DIR, which defaults to ~/.cache/sglang. Mount that directory and the Hugging Face cache on volumes that survive the pod. Then a restart reuses compiled kernels and downloaded weights.

The same field report defends the slow start. To them, 10 percent more tokens per second is worth more than a start that is 10 seconds faster. We agree for a fleet that runs for days. For a fleet that scales to zero every night, startup time is a real cost, and the persistent caches are the first fix.

Health checks: /ready, /health, and /health_generate

SGLang has two kinds of health endpoint:

• /ready returns 200 when the tokenizer manager is ready and 503 before that.
• /health and /health_generate share one handler. They return 503 while the server starts and while it shuts down.
• Otherwise the handler sends a one-token request with max_new_tokens 1 and waits for any output from the detokenizer.
• SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION defaults to true, so a plain /health also generates that token.
• The wait is SGLANG_HEALTH_CHECK_TIMEOUT, 20 s by default. With no output in that time, the handler marks the server UnHealthy and returns 503.

The docstring says that the probe request costs nothing when the server is busy, because any output from the scheduler counts as proof of life. That is true for the work, and false for the timing. A real incident: issue #22511 reports "health check failed. server could not get a response from detokenizer for last 20 seconds" on GLM-5 with 2 H800 GPUs under high load, on v0.5.10. A related issue, #35884, found that a timed-out probe left its request in the scheduler, so failed probes accumulated.

Our position: use /ready for the readiness probe. Keep the liveness probe from generating. Set SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION=false, and /health then returns 200 without a request to the scheduler. Use /health_generate for a slower deep check that alerts a person and restarts nothing. A busy server then does not lose its busiest pod to a probe that waited in the queue.

Graceful shutdown: signal the main process only

On SIGTERM, the HTTP server sets gracefully_exit, and /health starts to return 503. A watchdog in the tokenizer manager checks the flag every 5 s. It then logs the remaining requests every 5 s until none remain. Then it sends ShutdownReq to the schedulers and gives them 15 s to exit before it kills them. Figure 7.1 shows this path and the two paths that skip the drain.

We measured both cases on the Mac. Our capture kit stopped each server by sending SIGTERM to its whole process group. The scheduler child received the signal too and died first: Raw runs

[2026-10-16 14:01:45] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
[2026-10-16 14:01:45] Subprocess scheduler_0 (pid=5909) crashed with exit code -15. Triggering SIGQUIT for cleanup...
[2026-10-16 14:01:45] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
[2026-10-16 14:01:46] Sleeping 5 seconds before crash diagnostics to let GPU activity settle.

The parent saw a crashed child, took the crash path, and killed the process tree. No request was running at that moment, so nothing was lost. Under load, every running request would have failed. We then started a server, sent it a request for 3,000 output tokens, and sent SIGTERM to the main process only:

[2026-10-16 14:13:54] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
…
[2026-10-16 14:13:55] INFO:     127.0.0.1:53237 - "GET /health HTTP/1.1" 503 Service Unavailable
…
[2026-10-16 14:13:57] Gracefully exiting... Remaining number of requests 1. Remaining requests remaining_rids=['ef12359aaaaf40d78fedb4e7f7aaed25'].
…
[2026-10-16 14:14:04] INFO:     127.0.0.1:53209 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:14:07] Gracefully exiting... Remaining number of requests 0. Remaining requests remaining_rids=[].
…
[2026-10-16 14:14:07] INFO:     Finished server process [53853]

health before SIGTERM: 200
sent SIGTERM to the server process only
health during drain: 503
in-flight request: {"status": 200, "completion_tokens": 3000, "finish_reason": {"type": "length", "length": 3000}, "seconds": 14.2}
server exit code: 0

The request finished with all 3,000 tokens, and /health returned 503 during the drain. The server exited with code 0, thirteen seconds after the signal. docker stop and Kubernetes send SIGTERM to the main process of the container. That is the correct target when sglang serve is the main process, so start it with exec or as the entrypoint. The interactive command above makes bash the main process, and bash does not pass the signal on. A wrapper script that forwards the signal to a process group, or a supervisor that kills the group, gives you the first log instead.

The code has two exits that skip the drain. If the server is already UnHealthy, the watchdog exits at once. It also exits at once if the environment variable SGL_FORCE_SHUTDOWN is true. At v0.5.22 the watchdog reads that old name directly.

The environment module declares the new name SGLANG_FORCE_SHUTDOWN. Its compatibility code copies SGL_ variables to SGLANG_ names, never the other way. The code shows that setting only SGLANG_FORCE_SHUTDOWN does not reach the watchdog. We read this in the code and did not run it, so we call it a likely bug. If you need a forced stop, set SGL_FORCE_SHUTDOWN=1.

Our position: set the termination grace period longer than your longest request at your output cap. A 32K-token answer at 30 tokens per second needs more than 18 minutes. Cap max_tokens before you set the grace period, or a drain can never finish in time.

Before you trust a measurement

Three checks prevent most of the false results that we found on our own machines:

1. Flush the prefix cache before each run that you compare, and record the cached tokens, as in section 5.1. In our kit, a warm cache made a run at a higher load look faster. In a later test with Qwen/Qwen3-0.6B at a cap of 16, a repeat with no flush read a median TTFT about 4.8 times lower.
2. Record the SM clock and the temperature with every run, as in section 5.1. A hot L4 reported up to 28 percent less throughput.
3. On a shared GB10, run SGLang in a container with a memory limit, and log free memory during startup, as in section 3.2.

When a greedy output changes between two runs, check the cache first. A request that finds its prompt in the cache runs a different prefill path, as section 4.6 shows. One request at a time, a warm cache changed 21 of 32 of our greedy outputs, and a flush brought exact repeats back. These two tests ran on a shared L4 GPU with no clock lock, so trust the ratio and the counts, not the times.

The server aborts or hangs under heavy eviction

Symptom: in the middle of a multi-turn load, the server stops with CUDA error: an illegal memory access was encountered. Or the log stops with no error while the GPU stays at 100 percent.

We saw this on SGLang 0.5.22 with Qwen/Qwen3-0.6B, a pool of 8,192 tokens, and the trace of section 3.5. From the fourth turn, the pool evicted prefixes on almost every step. CUDA graphs and the overlap scheduler were on, which is the default. The runs were on the second GPU of our L4 node on 2026-10-23, which its owner shares and which had no clock lock.

variant                                          runs   aborts   hangs   clean
lru, CUDA graph and overlap on (default)         6      1        3       2
lru, CUDA_LAUNCH_BLOCKING=1                      3      0        0       3
lru, --disable-cuda-graph                        4      0        0       4
lru, --enable-hierarchical-cache                 3      1        0       2
lru, HiCache, CUDA_LAUNCH_BLOCKING=1             2      0        0       2

With the defaults, 4 of 6 runs failed. HiCache did not prevent it, and 1 of its 3 runs aborted. The abort and every hang stopped in the same place: the scheduler waited for the copy of a decode result to finish. With CUDA graphs off, 0 of 4 runs failed. With CUDA_LAUNCH_BLOCKING=1, 0 of 3 failed.

We did not find the kernel that faults, because the fault did not occur with blocking launches. Nothing was filed upstream.

Workaround: add --disable-cuda-graph. On our L4 with an 8B model, graphs saved about 1 percent of each decode step, as section 4.1 shows. A small model spends a larger share of each step on launches, so measure the cost on your model. CUDA_LAUNCH_BLOCKING=1 also hid the fault, but it makes every kernel launch wait for the one before it. Use it to debug, never to serve.

Also test: a larger pool. With a pool of 16,384 tokens or more, the same trace stopped thrashing, as section 3.5 shows. We did not count crashes at the larger pools, so this is not a proven fix.

The order to change settings

Most tuning advice lists flags. The order matters more than the list, because each step changes what the next step measures. Figure 7.1b shows the order this manual uses, and each row links to the section that measures it.

1. Measure with sglang.benchmark.serving at three concurrency levels, as in section 5.1. Without a baseline, no later change has a number.
2. Set --mem-fraction-static for the machine you share, as in section 3.2. On our Mac the automatic value crashed the server at concurrency 32.
3. Put shared text first in every prompt, as in section 3.4. One request id at the start of a system prompt cost us about 14 times the time to first token.
4. Cap --max-running-requests at the knee you measured. Past the knee on our L4, throughput fell while time to first token grew. Our GB10 kept rising to 64 users, and we did not measure a cap itself.
5. Only now change CUDA graph settings and --attention-backend, one change per run, as in section 4.1 and section 4.2. At v0.5.22 --disable-cuda-graph is deprecated in favour of --cuda-graph-backend-decode and --cuda-graph-backend-prefill.
6. Add tensor, data, or attention-data parallelism when one replica is saturated, as in section 6.1.
7. Split prefill from decode last, and only after you measured that prefill delays decode, as in section 6.3.

What production teams report

Three reports by teams outside SGLang show where operating effort goes.

LinkedIn, an independent engineering blog dated 2026-02-20, serves ranking models of 375M to 0.6B parameters on H100 GPUs. The workload needs only prefill and the logits of the last token. Their bottlenecks were the tokenizer, ZMQ messages, Python garbage collection, the GIL, and the scheduler, and none of them were GPU kernels.

Batch tokenization took P99 latency from 4,583 ms to 464 ms. A scoring path that skips decode took P99 from 6,220 ms to 454 ms. Garbage collection pauses of 100 to 300 ms went away with a warm-up and gc.freeze(). The 375M ranker went from 750 to 2,200 items per second per GPU. LinkedIn wrote most of the pull requests involved, so their "before" is an older SGLang.

CERIT-SC, a Czech academic service, moved from vLLM to SGLang in 2025 and reported 100 billion tokens served by 2026-04. Their traffic shape is the lesson. For Kimi K2.6 the median prompt was 2,179 tokens and the 95th percentile was 99,746 tokens, which they attribute to agentic coding tools. Their SGLang problems were tool calls that leaked into reasoning output, rare hangs, and model loops that held KV cache. Their fix was operational: an output cap of 32K tokens and a presence penalty for workloads that loop. They still run both engines, chosen per model.

Ant Group published a DeepSeek-R1 deployment on H20 GPUs with SGLang v0.5.2 on the LMSYS blog, dated 2025-09-26. They chose expert parallelism of 16 over 32 because at batch 32 EP16 gave 675 tokens per second against 585 for EP32. More speculative draft tokens gained 21% at batch 1 and lost about 18% at batch 32. They sell three latency tiers, and the strictest tier, below 30 ms TPOT, runs at batch 12. The measured tier table is a product decision, which is the point.

Our position: read a production report for its traffic shape and its order of fixes, never for its throughput number. LinkedIn fixed CPU work before GPU work. CERIT-SC capped output before it tuned anything. Ant Group chose the smaller expert group because the fault radius and the batch size it sells mattered more than peak throughput.