Serving

kllm serve starts a stock inference engine, SGLang or vLLM, on a plain or a sealed model. It builds the engine’s command line, adds a measured recipe where one applies, passes any further engine arguments through unchanged, and then replaces itself with the engine process. This page describes plain and sealed serving, engines, tensor parallelism, recipes, argument passthrough, dry runs, and kllm unlock for other launchers.

Synopsis

$ kllm serve MODEL [--engine {sglang,vllm}] [--tp N] [--host HOST] [--port PORT]
             [--key [KEYFILE]] [--key-source {passphrase,cdh}] [--production]
             [--int8 [PARTS]] [--client] [--recipe NAME] [--no-recipe] [--dry-run]
             [-- ENGINE_ARGS...]

Argument or option	Default	Meaning
MODEL	required	Model path or Hugging Face id, passed to the engine as given.
--engine	sglang	Inference engine to run: sglang or vllm.
--tp	see Tensor parallelism and GPUs	Tensor-parallel size.
--host	0.0.0.0	Address the engine binds.
--port	8000	Port the engine binds.
--key [KEYFILE]	absent	Serve a sealed model with its key. A bare --key selects ~/.kllm/keys/default.key.
--key-source	passphrase	passphrase: unlock the key file. cdh: inside a confidential VM, obtain the key from the Confidential Data Hub (kLLM Enterprise).
--production	off	Refuse keys from passphrases or the environment. Also enabled by KLLM_PRODUCTION.
--int8 [PARTS]	absent	MiniMax H3 only: run the diffusion transformer on INT8 kernels (attention, linear or all; a bare --int8 means all).
--client	off	Serve the cloud directory of a client-keyed model without its key.
--recipe NAME	none	Force a recipe by name.
--no-recipe	off	Use the engine’s defaults.
--dry-run	off	Print the engine command instead of running it.

Everything after the first -- goes to the engine unchanged; see Engine arguments.

Important

--key takes an optional value, so the model path must come first: kllm serve MODEL --key. In kllm serve --key MODEL, the model path is read as the key file.

Plain serving

Without --key or --client, kllm serve serves any model that the chosen engine supports. kLLM does not inspect or modify it.

$ kllm serve Qwen/Qwen3-8B
$ kllm serve ~/models/Qwen3-8B --engine vllm
$ kllm serve zai-org/GLM-5.3

The engine exposes its OpenAI-compatible API at http://HOST:PORT/v1:

$ curl http://localhost:8000/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Hello"}]}'

Note

kllm serve does not resolve model names such as qwen3:8b; MODEL is passed to the engine verbatim. Aliases are resolved by the commands of the upcoming 0.2 line, kllm pull and kllm run (Model library and naming); kllm run starts kllm serve on the pulled directory.

Sealed serving

$ kllm serve Qwen3-8B-sealed --key
$ kllm serve Qwen3-8B-sealed --key ~/.kllm/keys/qwen3.key --engine vllm

With --key, kllm serve:

1. unlocks the key file with its passphrase, read from KLLM_PASSPHRASE or from the terminal (see Model keys);

2. checks that the model is sealed and that its fingerprint matches the key, and refuses otherwise;

3. for a model with locked weights, writes an unlocked copy into the cache directory, or reuses the copy made earlier (Locked weights);

4. decrypts the permuted tokenizer from trusted.kllm into a new directory with mode 0700 under /dev/shm, together with copies of the model’s other non-weight files;

5. starts the engine on the sealed weights, unlocked, with that directory as its tokenizer (--tokenizer-path for SGLang, --tokenizer for vLLM).

The transform is read from the sealed model. The key itself is not passed to the engine. Clients use the same OpenAI-compatible API as for a plain model and exchange ordinary text; the permutation of token ids happens inside the engine’s tokenizer.

Warning

Outside a confidential VM, kllm serve --key decrypts the tokenizer into the host’s /dev/shm, writes the unlocked weights to the host’s file system, and the unlocked key is in host memory while kllm serve runs. The deployment then offers no protection from the host. Use this mode for development, or in a confidential VM with --key-source cdh; see Deployment profiles.

Warning

The decrypted tokenizer directory is not removed when the engine exits, because kllm serve replaces itself with the engine process. It remains in /dev/shm, as a directory named kllm-tok-*, until it is deleted or the machine restarts. Remove it after the server stops, or give each server its own memory-backed volume, as the Kubernetes example does.

Locked weights

By default, kllm seal locks the sealed weights: every weight is encrypted in place under a key derived from the model key (Locked weights). A locked model loads in any engine but computes nothing useful there, so kllm serve --key, kllm serve --key-source cdh and kllm unlock unlock the weights before the engine starts:

• The unlocked copy is written to ~/.cache/kllm/unlocked/NAME-DIGEST, where NAME is the sealed directory’s name and DIGEST identifies the key and the set of weight files. KLLM_UNLOCKED_DIR replaces ~/.cache/kllm/unlocked. The directory is created with mode 0700.

• The copy is made once per key and set of weight files and reused on later starts. A changed weight file (by size or modification time) leads to a new copy; the earlier one stays in the cache. Weight shards are decoded; the other files are hard-linked from the sealed directory where possible, or copied.

• The cache needs free space for one full copy of the weights. The research team measured unlocking at about 0.7 GB/s on a GPU and about 0.1 GB/s on CPUs.

• kLLM does not remove the copy. Delete it when the model is no longer served.

Important

The unlocked copy holds the sealed weights in the clear, as the GPU does while the model runs. A party that can read it and holds the public base can match the weights to it (Locked weights). Place the cache directory accordingly; on an ordinary host it belongs to the host.

A sealed MiniMax H3 pipeline is served with SGLang only, through sglang serve. kLLM builds a view of the sealed pipeline in /dev/shm that links the sealed files and holds the decrypted tokenizer files, and adds --model-variant from the sealed metadata. See Sealing and serving MiniMax H3. --int8 applies only to such a pipeline; it is a kernel optimization that is separate from sealing, described in Running the H3 diffusion transformer on INT8 kernels.

Engines

kLLM runs the engine installed in the same Python environment as kllm itself. Install the engine through the matching extra:

Engine	Extra	Command that kllm serve runs
SGLang	serve (sglang>=0.5)	python -m sglang.launch_server --model-path MODEL --tp N --host HOST --port PORT (plus --tokenizer-path DIR with --key)
vLLM	vllm (vllm>=0.11)	vllm serve MODEL --tensor-parallel-size N --host HOST --port PORT (plus --tokenizer DIR with --key)

A sealed MiniMax H3 pipeline runs sglang serve --model-path VIEW --num-gpus N instead. kLLM is not yet published on PyPI; install it from a clone of the repository, for example pip install -e '.[seal,serve]' or pip install -e '.[seal,vllm]'. See Installation guide and Serving with vLLM.

Tensor parallelism and GPUs

--tp sets the tensor-parallel size. Without it, kllm serve uses the recipe’s size if a recipe applies, and otherwise the number of visible GPUs, with a minimum of 1.

kLLM counts GPUs with nvidia-smi and honors numeric CUDA_VISIBLE_DEVICES values. A non-numeric value, such as GPU UUIDs, leaves every GPU counted. The first GPU’s name determines the GPU family that recipes are matched against. kllm serve prints the inventory on its first line, for example kLLM 0.1.0.dev0 · backend sglang · 8×H200.

Recipes

A recipe is a set of SGLang arguments measured for one model on one GPU family. kLLM 0.1.0.dev0 ships three. List them with kllm recipes.

Recipe	Model name	GPUs	Tensor-parallel size
glm-5.3-h200	glm-5.3	H200	8
glm-5.3-b300	glm-5.3	B300	8
glm-5.3-nvfp4-b300	glm-5.3-nvfp4	B300	8

• Recipes are consulted only for --engine sglang. vLLM always runs with its own defaults.

• A recipe matches when the last component of MODEL, lower-cased, equals its model name and the detected GPU family is one of its families. zai-org/GLM-5.3 and /models/GLM-5.3/ both match glm-5.3. If no GPU is detected, or the first GPU belongs to no known family, the family is not checked and the recipe still applies; use --no-recipe in that case if the recipe does not fit the hardware.

• Models without a matching recipe run with the engine’s defaults; kllm serve reports recipe  sglang defaults.

• --recipe NAME forces a recipe regardless of model and GPU, provided that the engine is SGLang and --no-recipe is not given. An unknown name fails with an error that refers to kllm recipes.

• --no-recipe disables recipes.

Recipe arguments are placed after kLLM’s own arguments and before passthrough arguments. See Contributing a serving recipe to contribute a measured recipe.

Engine arguments

Everything after the first -- is appended to the engine’s command line unchanged, after the recipe’s arguments, so that it takes precedence over them.

$ kllm serve Qwen/Qwen3-8B -- --context-length 32768
$ kllm serve Qwen3-8B-sealed --key --engine vllm -- --max-model-len 32768
$ kllm serve zai-org/GLM-5.3 -- --mem-fraction-static 0.80

Passthrough is accepted only by kllm serve; any other command given -- exits with an error. Use the engine’s own flag names, which differ between SGLang and vLLM.

Dry runs

--dry-run prints the command that kllm serve would run and exits. The interpreter and console-script paths are absolute and depend on the environment.

$ kllm serve Qwen/Qwen3-8B --dry-run -- --context-length 32768
kLLM 0.1.0.dev0 · backend sglang · 0×no GPU
  recipe  sglang defaults
  serving http://0.0.0.0:8000/v1 (OpenAI compatible)
/usr/bin/python3 -m sglang.launch_server --model-path Qwen/Qwen3-8B --tp 1 --host 0.0.0.0 --port 8000 --context-length 32768

Environment variables that a recipe or kLLM sets for the engine are printed before the command; the value of KLLM_MODEL_KEY is shown as <secret>.

Note

With --key, a dry run still unlocks the key and decrypts the tokenizer into /dev/shm, because the tokenizer path is part of the printed command.

Host, port and process

The engine binds --host (default 0.0.0.0) and --port (default 8000). kLLM adds no authentication and no TLS. Place the server behind a separate gateway, or bind 127.0.0.1.

kllm serve replaces itself with the engine process. Signals, exit codes and logs are those of the engine, so process supervisors and container runtimes manage it directly.

Client-keyed serving

kllm serve CLOUD_DIR --client serves the cloud directory written by kllm seal --client or kllm cloud, without a key and without a tokenizer. The weights of that directory are unlocked on the user’s side; a directory with locked weights is refused with a message that names kllm cloud. For a language model, the engine starts with --skip-tokenizer-init and exchanges token ids; for Z-Image, kLLM’s own latent server runs instead of an engine.

$ kllm serve Qwen3-8B-sealed-cloud --client --host 127.0.0.1
$ kllm serve Qwen3-8B-sealed-cloud --client --engine vllm --host 127.0.0.1

--client refuses --key, --key-source cdh and --production, and warns when the directory holds files that belong on the user’s side. See Client-keyed mode.

Warning

kllm serve --client binds 0.0.0.0 by default and has no authentication. Place it behind a separate gateway or tunnel, or bind 127.0.0.1.

Production mode and key sources

--production, or KLLM_PRODUCTION set to any value other than empty or 0, refuses passphrase and environment keys and requires --key-source cdh. Without it, kllm serve exits with --production takes the model key from the CDH only (--key-source cdh).

$ export KLLM_PRODUCTION=1
$ kllm serve /models/Qwen3-8B-sealed --key-source cdh --host 127.0.0.1 --port 8000

The cdh key source runs inside a confidential VM and is part of kLLM Enterprise. With the open-source package alone, --key-source cdh exits with key source 'cdh' needs kLLM Enterprise (kllm-enterprise). See Confidential VM deployment.

Other launchers: kllm unlock

$ kllm unlock MODEL [--key KEYFILE] [--engine {sglang,vllm}]

kllm unlock performs the key steps of sealed serving without starting an engine. It unlocks the key, checks the fingerprint, unlocks locked weights into the cache directory (Locked weights), decrypts the tokenizer into /dev/shm and prints only the tokenizer directory’s path on standard output; status lines go to standard error, among them weights unlocked: PATH for a locked model. A launcher such as NVIDIA Dynamo, llm-d or a Kubernetes manifest then passes the tokenizer path to the engine, and, for a locked model, the unlocked directory as the model:

TOK=$(kllm unlock Qwen3-8B-sealed 2>unlock.log)
MODEL=$(sed -n 's/^ *weights unlocked: //p' unlock.log)
MODEL=${MODEL:-Qwen3-8B-sealed}      # a model sealed with --no-lock is used as it is
python -m sglang.launch_server --model-path "$MODEL" --tokenizer-path "$TOK"
vllm serve "$MODEL" --tokenizer "$TOK"

Passing the sealed directory of a locked model to the engine starts it on the locked weights, which compute nothing useful.

For a MiniMax H3 pipeline, kllm unlock prints the directory of the in-memory pipeline view for sglang serve instead; pass it as --model-path together with the partition as --model-variant (for example fl2va).

The key file defaults to ~/.kllm/keys/default.key and --engine to sglang. For unattended starts, supply the passphrase through KLLM_PASSPHRASE. The warnings about /dev/shm and the cache of unlocked weights under sealed serving apply equally.

Note

The Dynamo, llm-d and Kubernetes examples in the repository have not been run end to end. See NVIDIA Dynamo and Kubernetes and llm-d.

See also

• Model keys for supplying the model key.

• Client-keyed mode for serving without the key in the cloud.

• Serving with vLLM for the differences under vLLM.

• Contributing a serving recipe for the recipe format.

• Troubleshooting for error messages.