Async, Hogwild and Disaggregated Inference

Share, split, route and offload the KV cache. Formulas, crossover numbers and the cases where each method loses.

Core ideas#

Prefill: one forward over the full prompt. Compute-bound.
Decode: one token at a time. Reads all weights. Memory-bandwidth-bound.
Hogwild!: N copies of one LLM write to one shared KV cache.
Emergent coordination: reasoning models divide the work with no fine-tuning.
RoPE: positions are rotations, so shared cache entries stay valid.
Disaggregation: prefill pool and decode pool. KV moves between them.
Cache-aware routing: send the request to the replica that holds its prefix.
KV offloading: KV cache leaves the GPU for CPU DRAM, then disk.

Factor	Hogwild!	Disaggregation	Cache-aware routing	LMCache
What it does	N workers, one shared KV cache	Prefill and decode on separate pools	Route on prefix-hash match	Cross-engine KV on CPU DRAM + disk
Wins when	Long reasoning, high p	Prompts >512 tokens + outputs >200 tokens	Prefixes repeat across replicas	KV exceeds HBM
Loses when	Short interactive chat	Small cluster (< 4 GPUs)	Cross-region RTT beats saved prefill	Short contexts (<1K tokens)
Number	1.8x at p = 0.7, N = 4	~6x DeepSeek-R1 on GB200 NVL72	~80 ms hit vs ~800 ms miss	3-5% overhead at low pressure
Status	Research, April 2026	Dynamo 1.0 GA, llm-d 0.5	vLLM Router, llm-d router	vLLM 0.11.0 async path

Setting	Default	Note
`CACHE_HIT_MS`	80	TTFT with the prefix resident
`CACHE_MISS_MS`	800	TTFT with a cold prefill
`REPLICAS_PER_REGION`	4	Three regions
`PREFILL_TOK_PER_MS`	40.0	Per GPU
`KV_BYTES_PER_TOKEN_70B_FP8`	125_000	KV size per token
`NIXL_RDMA_GB_S`	100	RDMA transfer rate
`NIXL_TCP_GB_S`	10	TCP fallback rate
`KV_BLOCK_TOKENS`	16	Tokens per KV block

def expected_speedup(T_serial: int, p: float, c: int, N: int,

steps_per_worker: int) -> float:

parallel = T_serial * ((1 - p) + p / N) + c * NAmdahl plus coordination

return T_serial / parallel

def ms_disaggregated(prompt: int, output: int, use_rdma: bool = True) -> float:

prefill_ms = prompt / PREFILL_TOK_PER_MS

kv_bytes = prompt * KV_BYTES_PER_TOKEN_70B_FP8

transport = NIXL_RDMA_GB_S if use_rdma else NIXL_TCP_GB_S

transfer_ms = (kv_bytes / 1e9) / transport * 1000the transfer tax

decode_ms = output / DECODE_TOK_PER_MS_DECODE_GPUfaster decode pool

return prefill_ms + transfer_ms + decode_ms

Task	Code
Hogwild! baseline, one worker	`run_hogwild(n_workers=1, step_budget=200, target_per_category=100, coordination_weight=0.8)`
Hogwild! with two workers	`run_hogwild(n_workers=2, step_budget=200, target_per_category=100, coordination_weight=0.8)`
Sweep coordination	`for cw in (0.0, 0.2, 0.5, 0.8, 1.0):`
Compare three routers	`for strategy in ("ROUND_ROBIN", "REGIONAL", "GLOBAL"):`
Colocated request time	`ms_colocated(prompt, output)`
Disaggregated over TCP	`ms_disaggregated(prompt, output, use_rdma=False)`
Compare offload configs	`for cfg in ("NATIVE_ONLY", "CPU_OFFLOAD", "LMCACHE"):`

Symptom	Cause	Fix
Prefix hit rate drops to 8%	Round-robin ALB	Cache-aware router on prefix hash
Replica refuses to start after failover	No tokenizer or quantization config	Three-file DR manifest
Disaggregated is slower	Short prompts, transfer tax	Colocate prompts < 512 tokens
Same 2K prompt re-prefilled	Preemption at 100% HBM	LMCache or native CPU offload
Cross-region route raises TTFT	440 ms round-trip	Keep routing regional
EU request served in us-east-1	Router ignores residency	Partition routers by residency
Hogwild! speedup below 1	Workers duplicate work	Use a reasoning model
Hogwild! gain under 1.3x	Coordination dominates	Revert to single-worker

KV transfer cost?: 20-80 ms for 4K-prompt KV on 70B FP8 over NIXL.
Hogwild! with spec decode?: The speedups multiply. 3x and 1.8x give 5.4x.
Why minimize prefill plus network?: A cross-region route adds a 440 ms round-trip to the request. Compare it with the prefill that a cache hit saves.
What breaks DR in 32% of cases?: Missing tokenizer or quantization configs.
Colocation waste?: 20-40% of GPU time on the wrong resource.