Kubernetes and llm-d

On Kubernetes a sealed model runs as an ordinary engine Deployment. The container first runs kllm unlock, which decrypts the tokenizer into a memory-backed /dev/shm and prints its path, and, for a model with locked weights, unlocks the weights into a cache directory. It then replaces itself with the engine, started on the unlocked weights with that tokenizer. The same wrapping applies to llm-d, which schedules vLLM pods behind its inference gateway. This page lists the prerequisites and describes the example Deployment, its SGLang variant and the llm-d setup.

Warning

Neither the manifest nor the llm-d setup on this page has been run on a cluster. They are taken from examples/kubernetes/. Review and test them before use.

Note

Planned — on the Roadmap, not available in 0.1.0.dev0. Dynamo and llm-d setups that run end to end and are kept in continuous integration, and container images published for each SGLang and vLLM release.

Prerequisites

An image. The manifest uses kllm:vllm, built from the repository’s Dockerfile with --build-arg BASE=vllm/vllm-openai:latest (see Docker). No image is published; push the image to a registry that the cluster can pull from, and replace image: with its full reference.
GPU scheduling. Nodes that advertise the nvidia.com/gpu resource.
The sealed model. A PersistentVolumeClaim named models that contains the sealed directory at Qwen3-8B-sealed. It is mounted read-only at /models.
The key. A Secret named kllm-key with two entries: model.key (the key file) and passphrase (its passphrase).

Deploying the example

Create the Secret and apply the manifest:

$ kubectl create secret generic kllm-key --from-file=model.key \
    --from-literal=passphrase=...
$ kubectl apply -f examples/kubernetes/kllm-vllm.yaml

The manifest defines a Deployment and a Service, both named qwen3-8b-sealed:

# A sealed model on Kubernetes with the kLLM vLLM image (see the Dockerfile).
# Development setup: the key file and its passphrase come from a Secret. In production the
# pod runs in a confidential VM and takes the key from the deployer's key broker
# (kLLM Enterprise).
#
#   kubectl create secret generic kllm-key --from-file=model.key \
#     --from-literal=passphrase=...
#   kubectl apply -f examples/kubernetes/kllm-vllm.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen3-8b-sealed
spec:
  replicas: 1
  selector:
    matchLabels: {app: qwen3-8b-sealed}
  template:
    metadata:
      labels: {app: qwen3-8b-sealed}
    spec:
      containers:
        - name: engine
          image: kllm:vllm
          command: ["sh", "-c"]
          # kllm unlock writes the secret tokenizer to /dev/shm (memory) and prints its path.
          # For locked weights (the default of kllm seal) it also unlocks them into
          # KLLM_UNLOCKED_DIR and reports "weights unlocked: PATH" on standard error; the
          # engine must load that directory. A model sealed with --no-lock is used as it is.
          args:
            - >-
              TOK=$(kllm unlock /models/Qwen3-8B-sealed --key /keys/model.key --engine vllm
              2>/dev/shm/unlock.log)
              && MODEL=$(sed -n 's/^ *weights unlocked: //p' /dev/shm/unlock.log)
              && exec vllm serve "${MODEL:-/models/Qwen3-8B-sealed}" --tokenizer "$TOK"
              --served-model-name Qwen3-8B --host 0.0.0.0 --port 8000
          env:
            - name: KLLM_PASSPHRASE
              valueFrom: {secretKeyRef: {name: kllm-key, key: passphrase}}
            - name: KLLM_UNLOCKED_DIR
              value: /unlocked
          ports: [{containerPort: 8000}]
          resources:
            limits: {nvidia.com/gpu: 1}
          readinessProbe:
            httpGet: {path: /health, port: 8000}
            periodSeconds: 10
          volumeMounts:
            - {name: models, mountPath: /models, readOnly: true}
            - {name: key, mountPath: /keys, readOnly: true}
            - {name: shm, mountPath: /dev/shm}
            - {name: unlocked, mountPath: /unlocked}
      volumes:
        - name: models
          persistentVolumeClaim: {claimName: models}
        - name: key
          secret: {secretName: kllm-key, items: [{key: model.key, path: model.key}]}
        - name: shm
          emptyDir: {medium: Memory, sizeLimit: 8Gi}
        # The unlocked weights, in the clear on the node; deleted with the pod, so every new
        # pod unlocks them again. Size it for the model's weights.
        - name: unlocked
          emptyDir: {sizeLimit: 20Gi}
---
apiVersion: v1
kind: Service
metadata:
  name: qwen3-8b-sealed
spec:
  selector: {app: qwen3-8b-sealed}
  ports: [{port: 8000, targetPort: 8000}]

Elements of the manifest
Element	Purpose
`command` and `args`	`sh -c` runs `kllm unlock ... --engine vllm`, stores the printed tokenizer path in `TOK`, reads the `weights unlocked:` line of its standard error into `MODEL`, and then `exec`s `vllm serve` on the unlocked weights with `--tokenizer "$TOK"`. A model sealed with `kllm seal --no-lock` reports no such line and is served from the sealed directory. `exec` makes vLLM the container’s main process, and `--host 0.0.0.0` makes it reachable on the pod’s address.
`KLLM_PASSPHRASE`	Supplies the key file’s passphrase from the Secret, since the container has no terminal.
`KLLM_UNLOCKED_DIR` and volume `unlocked`	The cache for the unlocked weights (Locked weights): an `emptyDir` with `sizeLimit: 20Gi` at `/unlocked`, because the `models` volume is read-only. Size it for the model’s weights. It is deleted with the pod, so every new pod unlocks the weights again.
Volume `key`	Mounts only the `model.key` entry of the Secret at `/keys/model.key`.
Volume `shm`	An `emptyDir` with `medium: Memory` and `sizeLimit: 8Gi` at `/dev/shm`. The decrypted tokenizer is written there, in node memory rather than on disk, and the volume is deleted with the pod. kLLM itself does not remove the tokenizer.
`readinessProbe`	Polls vLLM’s `/health` endpoint on port 8000 every 10 seconds.
Service	Exposes port 8000 inside the cluster. It adds no authentication.

Important

The unlocked weights are on the node, in the clear, and so is the decrypted tokenizer; the warning below applies to both. Starting the engine on the sealed directory of a model with locked weights, the default since weight locking was added, starts it on weights that compute nothing useful.

The engine serves the OpenAI-compatible API on port 8000 under the model name Qwen3-8B. To test it from a workstation:

$ kubectl port-forward service/qwen3-8b-sealed 8000:8000
$ curl http://127.0.0.1:8000/v1/models

Warning

This manifest is a development setup. The key file and its passphrase are stored in a Kubernetes Secret, which cluster administrators and anyone with read access to Secrets in the namespace can read, the decrypted tokenizer is in the node’s memory, and the unlocked weights are on the node. None of them is protected from the node or the cluster operator. In production the pod runs in a confidential VM and takes the key from the deployer’s key broker, which requires kLLM Enterprise; kllm unlock in 0.1.0.dev0 reads key files only and is refused in production mode (KLLM_PRODUCTION other than empty or 0). See Confidential VM deployment.

Running SGLang instead of vLLM

Use the kllm:sglang image and replace the container’s args with an SGLang launch. kllm unlock defaults to --engine sglang:

image: kllm:sglang
command: ["sh", "-c"]
args:
  - >-
    TOK=$(kllm unlock /models/Qwen3-8B-sealed --key /keys/model.key 2>/dev/shm/unlock.log)
    && MODEL=$(sed -n 's/^ *weights unlocked: //p' /dev/shm/unlock.log)
    && exec python3 -m sglang.launch_server --model-path "${MODEL:-/models/Qwen3-8B-sealed}"
    --tokenizer-path "$TOK" --served-model-name Qwen3-8B --host 0.0.0.0 --port 8000

--host 0.0.0.0 makes the server reachable on the pod’s address, which the Service and the readiness probe need. Keep the env, volumeMounts and volumes sections of the vLLM manifest, including the KLLM_UNLOCKED_DIR variable and the unlocked volume.

llm-d

llm-d schedules vLLM pods behind its inference gateway, with prefix-cache-aware and load-aware routing. A sealed model fits without changes to llm-d: in the model service’s container specification, use the kLLM image and wrap the vLLM command in the same way:

command: ["sh", "-c"]
args:
  - TOK=$(kllm unlock /models/sealed --key /keys/model.key --engine vllm
    2>/dev/shm/unlock.log) &&
    MODEL=$(sed -n 's/^ *weights unlocked: //p' /dev/shm/unlock.log) &&
    exec vllm serve "${MODEL:-/models/sealed}" --tokenizer "$TOK" --port 8000

Add the key Secret, the KLLM_PASSPHRASE variable and the memory-backed /dev/shm volume from kllm-vllm.yaml, and, for a locked model, the KLLM_UNLOCKED_DIR variable and a writable volume for the unlocked weights.

The choice of prefix-cache scorer determines which llm-d components hold the secret vocabulary:

Scorer	Requirement
Approximate prefix-cache scorer	Hashes prompt text and needs no tokenizer. Only the vLLM pods hold the unlocked tokenizer.
Precise prefix-cache scorer	Tokenizes in the scheduler. Give the scheduler the unlocked tokenizer, and keep the scheduler in the trusted domain, because it then holds the secret vocabulary.

Any component that holds the unlocked tokenizer can read requests and responses in clear text; place every such component in the trusted domain (Trusted domain) of the deployment profile in use (Deployment profiles).