Flags and defaults
These are the flags this manual changes or explains, with the default the code sets, the value our Mac resolved, and when to change it.
sglang serve --help lists 535 options at 0.5.22 (the captured help output). Most of them never matter to you. The table below holds the main ones this manual uses. The sections name a few more flags where they apply.
Many defaults are None in the code. A startup hook replaces them with a value that depends on the GPU, the model, and the free memory. The "On our Mac" column shows that resolved value, from the server_args line of the default server log.
Memory and admission
| Flag | Default in the code | On our Mac | Change it when | Section |
|---|---|---|---|---|
--mem-fraction-static | None, derived from GPU size at startup | 0.4, set by the kit | the machine runs other programs, or the server crashes on large batches | 3.2 |
--max-total-tokens | None, derived from the memory budget | 181,574 tokens in the pool | you want a fixed pool size for a test | 2.3 |
--max-running-requests | None, derived from the pool and the context length | 4,096 | you know your concurrency limit, or you turn on speculative decoding | 2.3 |
--chunked-prefill-size | None, derived from GPU memory | 4,096 | long prompts stall decode, or prefill is too slow | 2.2 |
--max-prefill-tokens | 16,384 | 16,384 | rarely | 2.2 |
--schedule-conservativeness | 1.0 | 1.0 | you see retractions in the log, raise it | 2.3 |
--page-size | None, derived per attention backend | 1 | the attention backend needs pages | 3.1 |
Scheduling
| Flag | Default in the code | On our Mac | Change it when | Section |
|---|---|---|---|---|
--schedule-policy | fcfs | fcfs | many queued requests share long prefixes, try lpm | 2.2 |
--disable-overlap-schedule | False | False | only to debug, and never on MLX at 0.5.22, where it crashed | 2.4 |
--stream-interval | 1 | 1 | you stream to many clients and want fewer messages | 2.5 |
Cache
| Flag | Default in the code | On our Mac | Change it when | Section |
|---|---|---|---|---|
--disable-radix-cache | False | False | only to measure the cache, as our kit does | 3.3 |
--radix-eviction-policy | lru | lru | a few prefixes are hot and many are cold | 3.5 |
--enable-hierarchical-cache | off | off | the working set of prefixes is larger than GPU memory | 3.5 |
--kv-cache-dtype | auto | auto | you need more context per GPU and can test the quality | 4.3 |
--enable-cache-report | False | True, set by the kit | always, so clients see cached_tokens | 3.4 |
Execution
| Flag | Default in the code | On our Mac | Change it when | Section |
|---|---|---|---|---|
--attention-backend | None, chosen per GPU and model | torch_native, which MLX does not use | you measured a better backend for your model | 4.2 |
--grammar-backend | None, which resolves to xgrammar | xgrammar | a grammar fails to compile | 4.5 |
--disable-cuda-graph, deprecated | False | True, as the Apple instructions say | on Apple silicon, where CUDA graphs do not exist. Prefer --cuda-graph-backend-decode disabled and --cuda-graph-backend-prefill disabled | 4.1 |
--speculative-algorithm | None | None | decode latency at low concurrency matters most | 4.4 |
--enable-deterministic-inference | False | False | you need the same output for the same input in any batch | 4.6 |
--disable-cuda-graph still works at 0.5.22, and the per-phase flags --cuda-graph-backend-decode and --cuda-graph-backend-prefill replace it.
Parallelism and operations
| Flag | Default in the code | On our Mac | Change it when | Section |
|---|---|---|---|---|
--tp-size | 1 | 1 | one GPU cannot hold the weights | 6.1 |
--dp-size | 1 | 1 | one copy of the model is saturated and you have GPUs for another | 6.1 |
--attn-dp-size | 1 | 1 | you serve an MLA model such as DeepSeek on many GPUs | 6.1 |
--ep-size | 1 | 1 | you serve a large MoE model | 6.2 |
--enable-metrics | False | True, set by the kit | always in production | 5.2 |
--watchdog-timeout | 300 s | 300 s | a single forward pass can take longer than five minutes | 7.1 |
SGLang 0.5.22 · baa9bef · 2026-10-13 · Edition 2026.10