Installation guide

kLLM 0.1.0.dev0 is installed from a source checkout or built into a container image on top of an SGLang or vLLM image. This page lists the requirements, describes both installation methods and the development installation, and shows how to verify the result.

Note

kLLM is not yet published on PyPI. pip install kllm becomes available with the first tagged release. Until then, use one of the methods below.

Requirements

Component	Requirement
Operating system	Linux. Windows and macOS installers are not provided.
Python	3.10 or later. Continuous integration tests Python 3.10 and 3.12.
GPU	An NVIDIA GPU supported by the chosen engine, for serving. kLLM detects GPUs with nvidia-smi; recipes are selected by GPU family.
Inference engine	SGLang 0.5 or later, or vLLM 0.11 or later, installed through the extras below. The published measurements used SGLang 0.5.21 and vLLM 0.31.0.
Disk	Space for the source checkpoint and the sealed checkpoint, which has the same tensors, shapes and dtypes as the original.

Sealing does not require a GPU for LLMs: Qwen3-30B-A3B-FP8 was sealed in 12 minutes on CPU. Sealing the MiniMax H3 FL2VA pipeline (135 GB) took about 15 minutes on one A40. Both figures were measured before weight locking was added (Sealing time).

Installing from source

Clone the repository and install it in editable mode with the extras for the tasks you need:

$ git clone https://github.com/ANetResearch/kLLM
$ cd kLLM
$ pip install -e '.[seal,serve]'

For vLLM instead of SGLang:

$ pip install -e '.[seal,vllm]'

Install kLLM into a virtual environment, the same one that holds the engine. kllm serve starts the engine from the interpreter it runs in: SGLang as python -m sglang.launch_server and vLLM through the vllm console script installed next to that interpreter.

Extras

The base package has no dependencies. Each extra adds the dependencies for one set of commands.

Extra	Installs	Enables
serve	sglang>=0.5	Serving with SGLang (the default engine)
vllm	vllm>=0.11	Serving with --engine vllm
keyed	cryptography>=42	Key files and tokenizer encryption (pulled in by seal)
seal	keyed, torch, safetensors, numpy	kllm keygen, kllm seal, kllm serve --key, kllm serve --client, kllm unlock, kllm verify, kllm cloud
client	seal, transformers, fastapi, uvicorn	kllm client proxy on the user’s machine
image	seal, diffusers>=0.36, transformers, fastapi, uvicorn, pillow	Z-Image serving and kllm client generate
int8	triton>=3, comfy-kitchen>=0.2.39	kllm serve --int8 for the MiniMax H3 diffusion transformer
dev	seal, pytest>=8, ruff>=0.6	Running the test suite and the linter

Without extras, kllm version, kllm recipes and kllm serve --dry-run work, and kllm serve can start a plain model if the engine is installed separately. Every keyed command, including kllm serve --client on a cloud host that holds no key, needs the seal extra; without it the command exits with status 2 and asks to install kllm[seal].

Typical combinations:

Machine	Command
Keyed serving host (SGLang)	pip install -e '.[seal,serve]'
Keyed serving host (vLLM)	pip install -e '.[seal,vllm]'
Sealing workstation, no engine	pip install -e '.[seal]'
User’s machine in client-keyed mode (LLM)	pip install -e '.[client]'
User’s machine in client-keyed mode (Z-Image)	pip install -e '.[client,image]'
MiniMax H3 host with INT8 kernels	pip install -e '.[seal,serve,int8]'

Verifying the installation

$ kllm version
kllm 0.1.0.dev0
$ kllm recipes

kllm recipes lists the built-in serving recipes: three, all for GLM-5.3 on H200 or B300 GPUs and SGLang only. To check the engine command line without starting a server, use --dry-run:

$ kllm serve Qwen/Qwen3-1.7B --dry-run

The command prints the GPU inventory, the recipe in use and the exact engine command, then exits.

Commands of the 0.2 line

The commands of the upcoming 0.2 line (Model, key and runtime commands (0.2 line)) are installed with the same source checkout. Two of them need packages that no extra installs yet:

• kllm pull downloads from Hugging Face with huggingface_hub and prints an installation hint when it is missing:

  $ pip install 'huggingface_hub>=0.24'
  

• ModelScope references (ms://) need the modelscope package.

kllm doctor checks the host: the driver and GPUs, disk space, /dev/shm, the server ports, the permissions of the key directory and the installed packages.

$ kllm doctor

Shell completion is printed by kllm completion for bash, zsh and fish.

Installing with Docker

The repository ships a Dockerfile that installs kLLM with the seal extra on top of an engine image and sets kllm as the entry point. Build it from the repository root:

$ docker build -t kllm:sglang .
$ docker build -t kllm:vllm --build-arg BASE=vllm/vllm-openai:latest .

Warning

The base images are not pinned. The default BASE is lmsysorg/sglang:latest, and the vLLM build above uses vllm/vllm-openai:latest, so two builds at different times can contain different engine versions. Pass a specific tag through --build-arg BASE= for reproducible images.

The image does not include the client, image or int8 extras, nor the benchmarks/, tools/ and examples/ directories. Container images are not yet published; publishing one for each SGLang and vLLM release is on the Roadmap.

Running the image is described in Docker.

Development installation

Contributors install the dev extra and run the tests and the linter:

$ uv venv --python 3.11 && uv pip install -e '.[dev]'
$ .venv/bin/pytest && .venv/bin/ruff check .

The tests run on CPU. See Contributing to kLLM.

Note

Planned — not available. Installers for Windows and macOS, and a published PyPI package installable with pip install kllm. See Roadmap.

Next steps

Continue with Quickstart.