Manuals / SGLang 101
00

How to read this manual

Install SGLang, serve your model, and then learn which few settings matter for your traffic, with the numbers we measured when we changed them.

This manual is for the engineer who wants an open model on their own GPU this week. It starts with a server that answers, and it ends with a fleet that you can measure and run.

These commands make an environment, install SGLang, start a server, and send it one request:

uv venv --python 3.12
source .venv/bin/activate
uv pip install --prerelease=allow "sglang==0.5.22"
sglang serve --model-path Qwen/Qwen3-8B --host 127.0.0.1 --port 30000 \
  --enable-cache-report --enable-metrics --mem-fraction-static 0.85
curl -s http://127.0.0.1:30000/generate -H 'Content-Type: application/json' \
  -d '{"text": "The capital of France is", "sampling_params": {"max_new_tokens": 12, "temperature": 0}}'

Section 1.1 runs them step by step, with the output to expect after each one and the fix for each common error. We ran the uv install and the server command on an NVIDIA L4 with SGLang 0.5.22. The Apple silicon steps come from our Mac runs, and the Docker command follows the SGLang install page. Every measured number in this manual names the machine that it came from: an NVIDIA L4, an NVIDIA DGX Spark, or the Mac.

What you can do after this manual

  • Install SGLang, serve a model, and get the first answer from both of its APIs in about ten minutes.
  • Size the KV pool for your GPU, and fix a server that runs out of memory.
  • Tune the server for your traffic: shared prompts, long documents, many users, or FP8.
  • Benchmark your own server, and watch it in production with its metrics.
  • Choose how to scale to many GPUs, and judge a published benchmark before you trust it.

Who this manual is for

You have called a model through an API, and now you want to serve an open model yourself. You know what a token and a GPU are. You have not tuned a serving engine before, and you want to know which settings matter before you pay for them.

sglang serve --help lists 535 options at version 0.5.22. The SGLang documentation describes each of them. This manual tells you which ones to change on your first day, and which defaults assume a machine that you do not have. It also shows what each change did on our GPUs.

How the parts are ordered

Part 1 starts a server and checks it. Part 2 follows one request through the processes and the scheduler. Part 3 is the KV cache, which decides how many users one GPU serves.

Part 4 makes each forward pass fast, and Part 5 measures a server. Part 6 spreads one model over many GPUs. Part 7 operates SGLang and closes with reinforcement learning. The Reference holds the lookup tables and how this manual was made:

PartWhat it covers
1 · StartInstall with uv or Docker, the first answer from both APIs, the health checks and startup lines, then prefill against decode and the four numbers.
2 · The Request Pathsglang serve, the scheduler loop and its policies, admission and retraction, the overlap scheduler, and the response path.
3 · The KV CacheBytes per token, startup sizing, RadixCache, prefix-friendly prompts, eviction and HiCache, and hybrid models.
4 · Each Forward PassForwardMode and CUDA graphs, attention backends, quantization, speculative decoding, structured outputs, and deterministic inference.
5 · Measuring a Serversglang.benchmark.serving, the /metrics endpoint, and how to read benchmarks that other people publish.
6 · One Server Becomes a FleetTensor, data, attention-data, and expert parallelism, MoE serving, prefill-decode disaggregation, and the model gateway.
7 · Production and TrainingDeploying and tuning in order, and SGLang as the rollout engine of Miles.
R · ReferenceThe flags, metrics, incidents, terms, and sources this manual uses, in one place.

Each section ends with an "Our position" part that says what to do, and a short list of actions. Read those first when you need an answer today. Read the rest when you want the reason.

Seven lessons go deeper into the ideas this manual applies. The serving internals lesson builds paged attention and continuous batching. The goodput lesson defines the metrics. The speculative decoding lesson builds EAGLE-3, and the quantization lesson covers the formats. The serving selection lesson, the inference optimization lesson, and the prompt caching lesson complete the set. This manual stays with SGLang.

Colour in figures

Each hue keeps one meaning in every figure:

bluethe SGLang server process: tokenizer manager, scheduler, detokenizer
violetthe model and its forward pass: kernels, attention backends, CUDA graphs
greenthe KV cache: pool slots, radix tree nodes, cached tokens
plummodel compute: prefill and decode steps
olivethe client and the network: HTTP, the OpenAI API, the benchmark client
indigostreams and messages: ZMQ messages and streamed tokens
amberlifecycle: request states and scheduler phases
tealdurable state: host memory and storage tiers, weights on disk
rosefailure: out of memory, retraction, aborts, crashes
greyoutside SGLang: the operating system, other programs, the GPU driver

Figure 0.1 teaches the arrow styles with exchanges from our capture.

Fig. 0.1reading a sequence figure in this manualsequence
Reading a sequence figure in this manual Six lifelines: the client script, the tokenizer manager, the scheduler, the model on MLX, the detokenizer manager, and macOS. Nine numbered rows use the eight arrow styles: POST /v1/chat/completions as a solid ink call, TokenizedGenerateReqInput as a dotted indigo ZMQ message, waiting to running as a dashed amber state change, a prefill of 1 new token with 2998 cached as a solid plum model call, the Metal memory limit of 51.8 GB as a dashed olive outside effect, the model files in the Hugging Face cache as a solid teal durable write, BatchTokenIDOutput as a dotted indigo message, 200 OK with cached_tokens 2998 as a dashed ink reply, and Insufficient Memory at concurrency 32 as a dashed rose failure. client Python script tokenizer manager scheduler event loop model MLX, Qwen3 detokenizer manager macOS Metal, disk 1 POST /v1/chat/completions a call · solid ink 2 TokenizedGenerateReqInput a ZMQ message · dotted indigo 3 request waiting to running a state change · dashed amber 4 prefill 1 new token, 2998 cached model compute · solid plum 5 Metal memory limit 51.8 GB an outside effect · dashed olive 6 9 model files to the HF cache a durable write · solid teal 7 BatchTokenIDOutput a ZMQ message · dotted indigo 8 200 OK, cached_tokens 2998 a reply · dashed ink 9 Insufficient Memory at c=32 a failure · dashed rose
Each arrow style marks one kind of exchange, and every label is a real name, message, or number from our recorded runs. Read each numbered row from the tail of the arrow to its head, and the note under each label names the style.

SGLang 0.5.22 · baa9bef · 2026-10-13 · Edition 2026.10