GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling
Three layers, not one. Karpenter provisions nodes dynamically (under one minute, 40% faster than Cluster Autoscaler). KAI Scheduler handles gang scheduling, topology awareness, and hierarchical queues — it prevents the 7-of-8 partial allocation trap where seven nodes wait and burn on one missing GPU. Application-level autoscalers (NVIDIA Dynamo Planner, llm-d Workload Variant Autoscaler) scale on inference-specific signals — queue depth, KV cache utilization — not CPU/DCGM duty cycle. The classic HPA trap is that DCGMFIDEVGPUUTIL is a duty-cycle measurement: 100% could be 10 requests or 100. vLLM pre-allocates KV cache memory, so memory never triggers scale-down. This lesson teaches you to compose the three layers and avoid the default Karpenter WhenEmptyOrUnderutilized policy that terminates running GPU jobs mid-inference. Diagram the three autoscaling layers (node provisioning, gang scheduling, application-level) and name the tool used at each layer. Explain why DCGMFIDEVGPUUTIL is the wrong HPA signal for vLLM and name two replacements (queue depth, KV cache utilization). Describe gang scheduling and the partial-allocation failure mode KAI Scheduler prevents (7 of 8 GPUs idle). Name the Karpenter consolidation policy (WhenEmptyOrUnderutilized) that terminates running GPU jobs and state the 2026 safe alternative. Your team ships an LLM-serving service on Kubernetes. You set up HPA with DCGMFIDEVGPUUTIL as the signal. The service pins at 100% utilization during business hours. HPA never scales up — it…
GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling: Three layers, not one. Karpenter provisions nodes dynamically (under one minute,…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.