Run vLLM on Kubernetes

You must run many vLLM servers on a cluster. When you finish this section, you can pick the layer that runs vLLM on Kubernetes.

Start with what one Deployment gives you

The vLLM documentation shows a plain Kubernetes setup. It has a volume for the weights, a secret for the Hugging Face token, a Deployment that runs vllm serve, and a Service. A small Helm chart in the vLLM repository packages the same pieces. It adds a job that downloads the model and an optional Kubernetes autoscaler.

Here is a minimal version in our own form, for one GPU and one model. We did not run it on a cluster. It needs a cluster where the NVIDIA device plugin offers nvidia.com/gpu. For a model with restricted access, add a Secret named hf-token with a token key. Create a Secret named vllm-api-key with a key key, so that the /v1 routes need the key.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen3-8b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-qwen3-8b
  template:
    metadata:
      labels:
        app: vllm-qwen3-8b
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.31.0
          args: ["Qwen/Qwen3-8B", "--max-model-len", "8192", "--reasoning-parser", "qwen3"]
          ports:
            - containerPort: 8000
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
                  optional: true
            - name: VLLM_API_KEY
              valueFrom:
                secretKeyRef:
                  name: vllm-api-key
                  key: key
          resources:
            limits:
              nvidia.com/gpu: 1
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
            failureThreshold: 90
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 30
            failureThreshold: 3
          volumeMounts:
            - name: hf-cache
              mountPath: /root/.cache/huggingface
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: hf-cache
          emptyDir: {}
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen3-8b
spec:
  selector:
    app: vllm-qwen3-8b
  ports:
    - port: 8000
      targetPort: 8000

The image starts vllm serve, so args holds only the model and the flags. The startup probe allows 90 checks 10 seconds apart, which is 15 minutes. Our DGX Spark took about 8 minutes for its first start, so set the limit from your own measured cold start. The probes stop Kubernetes from sending traffic or restarting the pod while the weights load.

Replace the emptyDir cache with a PersistentVolumeClaim, so a new pod does not download the weights again. The memory volume at /dev/shm gives the processes shared memory, as --ipc=host does with Docker. The API key does not protect /health or /metrics, so also limit which pods can reach the Service with a NetworkPolicy.

This is enough for a few replicas of one model. A plain Service spreads requests at random, so a user's next turn can go to a replica that never cached the earlier turns. An autoscaler on CPU use also scales on the wrong signal, because GPU load is not CPU load.

Pick a layer for the features you need

llm-d published its version 1.0 on 2026-10-20. It joined the CNCF as a sandbox project in March 2026. Its router follows the Gateway API Inference Extension, a Kubernetes project for routing to model servers.

Know which vLLM features the layers use

The layers read the same vLLM interfaces. Check that your vLLM version supports the ones your layer needs:

• Metrics. The routers read vllm:num_requests_waiting, vllm:num_requests_running, and vllm:kv_cache_usage_perc to pick a replica.
• KV cache events. With --kv-events-config, vLLM publishes each block it stores and removes. A router uses these events to send a request to the replica that holds its prefix.
• KV connectors. The prefill and decode split and the shared cache use --kv-transfer-config.
• Health. /health serves the startup, liveness, and readiness probes.

Not measured on our hardware. We have no cluster run, and we did not apply the manifest above.