Cheat sheets

GPU Autoscaling on Kubernetes

Nodes with Karpenter, gangs with KAI Scheduler, replicas on queue depth, cache-aware routing, event storyboards.

  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Tasks#

TaskCode
Stop GPU job evictionconsolidationPolicy: WhenEmpty
Consolidate empty nodes after an hourconsolidateAfter: 1h
Avoid the eviction defaultnot WhenEmptyOrUnderutilized
Do not scale on duty cyclenot DCGM_FI_DEV_GPU_UTIL
Keep one replica readymin_workers=1
Boot nodes with weights on NVMesnapshot ID in EC2NodeClass
Route on cached prefixkv.cache.block_added
Compare the three scalerssimulate(strategy, reqs)

Safe disruption for GPU pools#

Karpenter NodePool, GPU pool
disruption:
consolidationPolicy: WhenEmptynever evict a running job
consolidateAfter: 1hconsolidate empty nodes after an hour

Three autoscaling layers#

provisionadd nodeall or nothingplaceconsumescalepublishclientPending podsneed 8 H100sprocessKarpenterNodePool, ~45-60sgpuGPU nodeNVLink domainprocessKAI Schedulergang, topology, queuesprocessDynamo Planner / llm-dWVAreplicasqueueSignalsqueue depth, KV utilizationmodelvLLM replicasserve requestsprovisionadd nodeall or nothingplaceconsumescalepublishclientPendingpodsneed 8 H100sprocessKarpenterNodePool,~45-60sgpuGPU nodeNVLink domainprocessKAISchedulergang,topology,queuesprocessDynamoPlanner /llm-d WVAreplicasqueueSignalsqueue depth,KVutilizationmodelvLLMreplicasserverequests

Scaling strategies in main.py#

FactorDUTY_CYCLEQUEUE_DEPTHKAI_GANG
SignalDCGM_FI_DEV_GPU_UTILlen(queue)len(queue)
Scale up whenutil > TARGET_GPU_UTILqd > 5qd > 3
Step upreplicas_target += 1max(1, qd // 5)max(2, qd // 3)
Scale down whenutil < 20qd == 0qd == 0
ResultDrops requestsReacts to the backlogAvoids partial-alloc stalls

Simulator constants#

SettingDefaultNote
NODE_PROVISION_SEC50Karpenter ~45-60s
CLUSTER_AUTOSCALER_SEC110Slower comparison
MODEL_LOAD_SEC4570B weights plus engine init
HPA_TICK_SEC15Control loop interval
TARGET_GPU_UTIL70Duty-cycle target
MAX_REPLICAS16Replica ceiling
CACHE_HIT_MS80TTFT with prefix resident
CACHE_MISS_MS800TTFT on cold prefill

Storyboard stages#

1ParseParse([]byte)(Bundle,error) checks observedAtand both items arrays.2GroupGroup(Bundle)([]Row,error) keeps max count perevent UID, then sums.3Connectwalks ownerReferences todeployment.kubernetes.io/revision.4ExportExport(Bundle)(Pack,error) writes schemaVersion1 with redactions.5parsePackparsePack(text) checks version, counts and timeorder.6timelinetimeline(pack) returns escaped HTML with aWarning filter.

Cache-aware routing#

1Routerhashes the prefix, for example the first 512tokens.2Replicaspublish KV-cache events as they allocate andevict blocks.3vLLM Routerindexes kv.cache.block_added into prefix hash toreplica.4Routersends the request to the replica with the match.5Routerwith no match, picks the least queue depth.6Global routeradds RTT to the hit cost. It stays local whencost passes CACHE_MISS_MS.7Residencypartition routers by residency boundary beforecache.

Troubleshooting#

SymptomCauseFix
HPA never scales at 100%Duty cycle hides 10 or 100 requestsScale on queue depth or KV utilization
Memory HPA never scales downvLLM pre-allocates KV cacheDrop memory as a signal
7 of 8 pods wait and burn moneyNo gang schedulingKAI Scheduler, all or nothing
Running inference evictedWhenEmptyOrUnderutilizedWhenEmpty plus consolidateAfter: 1h
Request times out at 2 a.m.Node provisioning took 3 minutesKarpenter plus a warm pool
Cache hit rate drops to 8%Round-robin load balancerCache-aware router
Replica refuses to start in DRTokenizer or quant config not syncedBack up the three-file DR manifest
One HPA for prefill and decodeTwo pod classes, two triggersPer-role HPA on separate Services

Event storyboard project#

python3 scripts/project_test.py kubernetes-event-storyboard --init /tmp/kubernetes-event-storyboard-work
python3 scripts/project_test.py kubernetes-event-storyboard --stage 1 --path /tmp/kubernetes-event-storyboard-work --strict
python3 scripts/project_test.py kubernetes-event-storyboard --all --solution --strict
cd projects/kubernetes-event-storyboard/solution
go run . --input fixtures/rollout.json --out /tmp/context-pack.jsonGo pack
node --experimental-strip-types cli.ts /tmp/context-pack.json /tmp/timeline.htmlHTML
kubectl get events -A -o jsonexport your own snapshot

Storyboard errors#

CommandWhat it does
bundle exceeds one MiBInput is over 1<<20 bytes
trailing JSONExtra data after the bundle
observedAt must be RFC3339Bad capture timestamp
invalid occurrence range or countLast before first, or count below 1
event UID changed identityUID reused for another object or reason
invalid or duplicate workload UIDEmpty or repeated workload UID
invalid context packTypeScript rejects the Go pack

Q&A#

Karpenter or Cluster Autoscaler?
Karpenter is ~45-60s. Cluster Autoscaler is ~90-120s for GPU nodes.
Which signal for prefill pods?
Queue depth. Decode pods scale on KV cache pressure.
Route across regions?
Rarely. 440 ms round trip beats the 800 to 80 ms saving.
What does round-robin cost?
Cache hit is ~80 ms, miss is ~800 ms. A 10x gap.
Two snapshots with counts 2 and 4?
One series. The count is 4, not 6.
Two revisions found?
The revision stays unknown. Never guess from names.

Tips#

  • Back up tokenizer.json or tokenizer.model with the weights.
  • Back up quantize_config.json and vllm_config.yaml.
  • Run a DR drill quarterly.
  • Run KAI as a secondary scheduler. Annotate workloads to use it.
  • Keep a tensor-parallel workload on one NVLink domain.
  • Give min_workers=1-2 to live chat and voice paths only.