Docker

The repository ships a Dockerfile that installs kLLM on top of an inference-engine image. The resulting image runs the kllm command as its entry point, so every argument after the image name is a kllm subcommand. This page describes how to build the image, what it contains and how to run it for plain, keyed and client-keyed serving.

Note

No container image is published for kLLM 0.1.0.dev0, and the project’s continuous integration does not build the Dockerfile. Build the image locally as described below. Publishing an image for each SGLang and vLLM release is on the Roadmap; it is not available in 0.1.0.dev0.

Building the image

Build from the repository root. The default base image is SGLang; pass BASE to build on vLLM instead:

$ docker build -t kllm:sglang .
$ docker build -t kllm:vllm --build-arg BASE=vllm/vllm-openai:latest .
Build arguments

Argument

Default

Meaning

BASE

lmsysorg/sglang:latest

The engine image that kLLM is installed into. It must provide python3 with pip and the engine itself (SGLang or vLLM).

Warning

The base images are not pinned. Both lmsysorg/sglang:latest and vllm/vllm-openai:latest move as new engine releases appear, so two builds made at different times can contain different engine versions, and a build does not reproduce the engine versions behind the published measurements (SGLang 0.5.21 and vLLM 0.31.0). For a reproducible image, pass a specific tag or digest, for example --build-arg BASE=<registry>/<image>:<tag>, and record it alongside the image.

Contents of the image

The build performs three steps on top of BASE:

  1. It copies pyproject.toml, README.md, LICENSE and src/ to /opt/kllm.

  2. It runs python3 -m pip install --no-cache-dir "/opt/kllm[seal]". The seal extra adds cryptography, torch, safetensors and numpy; the engine comes from the base image, not from the serve or vllm extras.

  3. It sets ENTRYPOINT ["kllm"] and CMD ["--help"]. Running the image without arguments prints the kllm help text.

The following are not in the image:

  • The client, image and int8 extras. kllm client proxy, kllm client generate, client-keyed Z-Image serving and kllm serve --int8 need a derived image.

  • The benchmarks/, tools/, examples/, docs/ and tests/ directories, which .dockerignore excludes.

To add an extra, derive an image from the one built above and install the extra from the copied source tree, which is the same mechanism the Dockerfile uses:

FROM kllm:sglang
RUN python3 -m pip install --no-cache-dir "/opt/kllm[seal,int8]"

Running a plain model

A plain model needs no key. The following command serves a checkpoint from the host’s ~/models directory on port 8000:

$ docker run --gpus all --shm-size 32g -v ~/models:/models -p 8000:8000 \
    kllm:sglang serve /models/Qwen3-8B

The options have the following roles:

Option

Role

--gpus all

Exposes the host’s NVIDIA GPUs to the container. kllm serve counts the visible GPUs and uses that count as the tensor-parallel size unless --tp or a recipe sets it.

--shm-size 32g

Enlarges the container’s /dev/shm. The engines use shared memory between their processes, and a keyed launch writes the decrypted tokenizer there.

-v ~/models:/models

Mounts the model directories. Paths given to kllm refer to the container’s file system.

-p 8000:8000

Publishes the engine’s port. kllm serve binds 0.0.0.0:8000 by default, which is reachable through port publishing.

On an image built on vLLM, pass --engine vllm; the default engine is sglang regardless of the base image:

$ docker run --gpus all --shm-size 32g -v ~/models:/models -p 8000:8000 \
    kllm:vllm serve /models/Qwen3-8B --engine vllm

Arguments after -- are passed to the engine unchanged. To inspect the engine command without starting it, add --dry-run:

$ docker run --rm -v ~/models:/models kllm:sglang serve /models/Qwen3-8B --dry-run \
    -- --context-length 32768

Creating a key and sealing in a container

kllm keygen and kllm seal read the key passphrase from KLLM_PASSPHRASE when it is set and non-empty, and otherwise from the terminal. Run them with -it for an interactive prompt, and mount ~/.kllm so that the key file is written to the host. Inside the container the default key path is /root/.kllm/keys/default.key.

$ docker run --rm -it -v ~/.kllm:/root/.kllm kllm:sglang keygen
$ docker run --rm -it -v ~/.kllm:/root/.kllm -v ~/models:/models \
    kllm:sglang seal /models/Qwen3-8B -o /models/Qwen3-8B-sealed --transform perm

Sealing does not need a GPU. See Model keys and Sealing for the options of both commands.

Serving a sealed model with a key

The usage comment in the Dockerfile serves a sealed model with the default key file and a passphrase taken from the host environment:

$ export KLLM_PASSPHRASE=...
$ docker run --gpus all --shm-size 32g -v ~/models:/models -v ~/.kllm:/root/.kllm \
    -e KLLM_PASSPHRASE -p 8000:8000 kllm:sglang serve /models/Qwen3-8B-sealed --key

Warning

This is a development setup, not a deployment profile. The key file and its passphrase are on the host, and kllm serve --key decrypts the tokenizer into the container’s /dev/shm, which is host memory. kLLM does not remove that copy. Outside a confidential VM this mode offers no protection from the host. See Deployment profiles and Confidential VM deployment.

Serving in client-keyed mode

In client-keyed mode the container receives only the -cloud directory that kllm seal --client writes; the key and the tokenizer stay with the user, who runs kllm client proxy on their own machine (see Client-keyed mode).

$ docker run --gpus all --shm-size 32g -v ~/models:/models -p 127.0.0.1:8000:8000 \
    kllm:sglang serve /models/Qwen3-8B-sealed-cloud --client

Warning

kllm serve --client has no authentication. Inside the container it binds 0.0.0.0 so that port publishing works; restrict exposure on the host side instead, as -p 127.0.0.1:8000:8000 does above, and reach the port through a tunnel or a gateway that adds authentication.

Client-keyed Z-Image runs kLLM’s own latent server, which needs the image extra. The image built from the unmodified Dockerfile serves client-keyed language models only.

See also