Troubleshooting

This page lists the error messages of the kllm command, grouped by task, with their causes and remedies. Messages are quoted as the code prints them, without the status symbols that precede some of them; <...> marks a value that varies. Errors that come from the engine itself are covered at the end.

Most errors exit with status 1. Errors of the sealed-model and key-source checks, missing extras and unknown recipes exit with status 2.

General diagnosis

Run kllm version to confirm the installed version (kllm 0.1.0.dev0).
Run kllm serve MODEL ... --dry-run to print the engine command without starting it, then run that command directly to see the engine’s own output. With --key, the dry run still unlocks the key and writes the decrypted tokenizer to /dev/shm.
Use full option names. Abbreviations such as --trans are accepted by the parser but are not documented.
Put the model before --key: kllm serve MODEL --key [KEYFILE]. Because --key takes an optional value, kllm serve --key MODEL reads MODEL as the key file and then reports that the model argument is missing.

Installation and extras

Message	Cause and remedy
model keys need kLLM's keyed extras (<module> is missing); install `kllm[seal]`	A command that handles keys or sealed models (`keygen`, `seal`, `serve --key`, `serve --client`, `unlock`, `client`) runs without the `seal` extra. From a source checkout, run `pip install -e '.[seal]'`. The cloud host of client-keyed serving also needs it.
`ModuleNotFoundError` for `diffusers` or `PIL` in `kllm client generate`	The `client` extra lacks the image dependencies. Install `pip install -e '.[client,image]'`.
`ModuleNotFoundError` for `sglang` or a missing `vllm` executable	The engine is not installed in the interpreter that runs `kllm`. Install the `serve` or `vllm` extra into the same environment.
`int8_linear takes bf16 activations, got <dtype>`	The INT8 linear kernels run on bf16 activations only. Serve H3 in bf16, or use `--int8 attention`.

kllm is not yet published on PyPI; see Installation guide.

Keys and passphrases

Message	Cause and remedy
`no terminal to read the key passphrase; set KLLM_PASSPHRASE`	The command runs without a terminal (a script, a container, a service). Set `KLLM_PASSPHRASE` for development starts. Do not use it in production.
`passphrases do not match`	`kllm keygen` asks twice; the entries differ. Run it again.
`passphrase must be at least 10 characters`	Choose a passphrase of 10 characters or more.
`<path> exists; pass --force to overwrite`	`kllm keygen` refuses to replace a key file. Choose another path with `-o`, or pass `--force` if the old key is no longer needed. Models sealed under the old key cannot be served without it.
no key at <path>; create one with `kllm keygen`	No key file at the default path `~/.kllm/keys/default.key` or at `--key`.
`<path> is not a kLLM v1 key file`	The file is not a kLLM key file, or was written by an incompatible version.
`wrong passphrase (or the key file was modified)`	The passphrase does not decrypt the key file, or the file was altered.

See Model keys.

Sealing

Message	Cause and remedy
`<output> exists`	The output directory, or with `--client` the cloud directory, already exists. Choose a new path or remove the old one.
`<src> is already sealed`	The source is a sealed model. Seal the original checkpoint instead.
`<architecture> is not supported yet (supported: Qwen3ForCausalLM, Qwen3MoeForCausalLM, Qwen2ForCausalLM, LlamaForCausalLM)`	The architecture cannot be sealed. It can still be served plain. See Supported models.
`MLP biases are not supported yet`	The checkpoint sets `mlp_bias`.
`partial rotary embeddings are not supported yet`	`partial_rotary_factor` is not 1.0.
`<method> checkpoints are not supported (FP8 is)`	A quantization method other than FP8, for example AWQ or GPTQ.
`no .safetensors files in <src>`	The sealer reads safetensors weights only. Convert `.bin` weights first.
`tokenizer vocabulary ids are not contiguous from 0`, `an added token sits inside the regular vocabulary`	The `tokenizer.json` does not have the layout the vocabulary permutation needs.
`FP8 checkpoints need --transform perm (orth would requantize)`	Add `--transform perm`. Other FP8 messages are listed in Sealing FP8 and mixture-of-experts checkpoints.
`--client supports LLMs and Z-Image pipelines`	`--client` on a pipeline other than Z-Image, such as MiniMax H3.
`attention biases are not supported yet`	The text encoder of an H3 or Z-Image pipeline has attention biases.
`only Z-Image's 2×2 patches (frame patch 1) are supported`, `image-conditioned Z-Image variants are not supported`	The Z-Image variant cannot be sealed for client-keyed serving.
`<class> is not supported yet (supported: Z-Image)`	A diffusers pipeline that is neither MiniMax H3 nor Z-Image.
`don't know how to transform <tensor>`	The checkpoint contains a tensor the sealer does not recognize. The sealer refuses rather than copy it unchanged.

Serving

Message	Cause and remedy
<model> is not a sealed model; seal it first with `kllm seal`	`--key` was given for a plain model. Omit `--key`, or seal the model.
`<model> was sealed with key <fingerprint>, not <fingerprint>`	The key file is not the one the model was sealed with. `kllm keygen` prints a key’s fingerprint when it creates the key.
`<path> is corrupted or was modified`, `<path> is not a kLLM trusted blob`, `<path> has trailing data`	The encrypted tokenizer `trusted.kllm` is damaged. Copy the sealed model again.
`<model> is locked with <version>; this kLLM reads v1`	The weights were locked by a kLLM version with another lock format. Use a matching kLLM version, or seal the original checkpoint again.
`<model> is client-keyed; serve it with --client`	A client-keyed Z-Image pipeline was served with `--key`. Serve its `-cloud` directory with `--client`.
`<model> uses scheme 'd1-v1'; reseal it with this kLLM`	A Z-Image pipeline sealed without `--client`. `kllm serve` does not serve this scheme; reseal the original pipeline with `--client`. See Generating images in client-keyed mode.
`MiniMax H3 pipelines are served with SGLang`	`--engine vllm` with an H3 pipeline. Omit `--engine`.
`--int8 applies to sealed MiniMax H3 pipelines (served with --key)`	`--int8` with any other model. See Running the H3 diffusion transformer on INT8 kernels for plain H3.
unknown recipe '<name>'; see `kllm recipes`	`--recipe` names no built-in recipe. `kllm recipes` lists the three recipes.
`--` passthrough is only valid for `kllm serve`	Arguments after `--` were given to another command.
`--production takes the model key from the CDH only (--key-source cdh)`, `KLLM_PRODUCTION=1 takes the model key from the CDH only`	Production mode (`--production` or `KLLM_PRODUCTION` set to a value other than empty or `0`) refuses passphrase keys. Unset `KLLM_PRODUCTION` for development.
`key source 'cdh' needs kLLM Enterprise (kllm-enterprise)`	`--key-source cdh` needs a key source that only kLLM Enterprise provides. See Confidential VM deployment.

If kllm serve picks the wrong tensor-parallel size, pass --tp. GPU detection uses nvidia-smi and resolves only numeric CUDA_VISIBLE_DEVICES values; with GPU UUIDs it counts every GPU.

Client-keyed mode

Message	Cause and remedy
`--client serves without a key; the key stays with the user`	`--client` together with `--key`, `--key-source cdh` or production mode.
<model> is not sealed; seal it with `kllm seal --client`	`--client` with a plain model.
<model> is locked; export the cloud's copy with `kllm cloud <model> -o DIR --key KEYFILE` on your side	The served directory has locked weights, for example a catalog model or the user’s sealed directory. Write the cloud’s copy with `kllm cloud` on the user’s machine and serve that directory.
`<model> is locked; unlock it for the cloud with --key`	`kllm cloud` without `--key` for a model with locked weights. Pass the key file the model was sealed with.
`--client serves sealed LLMs and client-keyed Z-Image`	`--client` with an H3 pipeline or a Z-Image pipeline sealed without `--client`.
Warning: <n> files here belong on the user's side (...); upload the -cloud directory of `kllm seal --client` instead	The served directory contains the tokenizer or `trusted.kllm`. Serve the `-cloud` directory and remove the user’s files from the cloud host.
`<model> is not a client-keyed pipeline; seal it with --client`	`kllm client generate` with a directory that was not sealed with `--client`. Pass the user’s sealed directory, not the `-cloud` one.
HTTP 400 `logprobs, prompt_logprobs, echo need the vllm engine; use /generate`	These fields of `/v1/completions` are available through the proxy with vLLM only.
HTTP 400 `/generate needs the sglang engine`, `streaming /generate is not supported; use /v1/completions`	The proxy forwards `/generate` to SGLang only, and without streaming.
HTTP 400 `need 1 to 512 token ids below <vocab>`	The Z-Image prompt encodes to no token ids or to invalid ids.
HTTP 400 `size must look like 1024x1024, got '<size>'`	`--size` is not of the form `WIDTHxHEIGHT`.

Known limitations of the proxy in 0.1.0.dev0:

Non-streaming requests block the proxy for the whole generation, so concurrent non-streaming clients are served one at a time. Streaming requests are not affected.
A non-streaming completion whose last token ends inside a multi-byte character returns empty text. Request a few more tokens, or use streaming.
A connection error to the cloud is returned as HTTP 500 without detail. Check the URL and the tunnel.
kllm client sends no credentials, so an engine started with its own --api-key cannot be used through it.

Engine errors and output problems

kllm serve replaces itself with the engine process, so errors after the line serving http://... come from SGLang or vLLM. Consult the engine’s documentation, and pass engine options after --.

The sealed model produces unrelated text in random scripts: The engine runs the sealed weights with a tokenizer other than the decrypted, permuted one, for example a launcher that loads the public tokenizer. Start sealed models with kllm serve MODEL --key, or pass the directory printed by kllm unlock as the engine’s tokenizer.
The sealed model produces meaningless output with the right tokenizer: The engine loads the locked weights of the sealed directory instead of the unlocked copy. kllm serve --key handles this itself; with another launcher, pass the directory that kllm unlock reports as weights unlocked: PATH on standard error as the engine’s model (Other launchers: kllm unlock).
The engine cannot find a tokenizer for a sealed model: A sealed LLM directory contains no plaintext tokenizer. Use kllm serve --key or kllm unlock, as above.
Perplexity differs between runs under vLLM: Start vLLM with --no-enable-prefix-caching for perplexity measurements. In the published client-keyed measurements, two of four vLLM launches differed from the others by 0.05 in mean |Δ log-prob|, while each launch was deterministic; see Client-keyed mode.
The disk fills up after serving locked models: kllm serve --key and kllm unlock write one unlocked copy of the weights per key and set of weight files to ~/.cache/kllm/unlocked (or KLLM_UNLOCKED_DIR), and do not remove it. Delete copies of models that are no longer served; they hold the weights in the clear. Point KLLM_UNLOCKED_DIR at a volume with room for the weights.
/dev/shm fills up: kllm serve --key and kllm unlock write the decrypted tokenizer, and for H3 a pipeline view, to a new /dev/shm/kllm-tok-* directory on every start, and do not remove it. Remove directories that are no longer in use; they contain the decrypted tokenizer. In a container, give /dev/shm enough space (--shm-size).
kllm demo up fails with AttributeError: kllm demo up is under repair in 0.1.0.dev0 and does not start.

If a problem is not listed here, search the issue tracker and report it with the output of kllm version, the command, its --dry-run output and the engine version.