Architecture overview

This page describes how the kLLM source tree is organized and how a command travels through it. It is the starting point for adding a model family, an engine, a recipe or a key source.

kLLM is a layer on top of the inference engine. It never runs a model itself: it rewrites checkpoints ahead of time and then starts a stock SGLang or vLLM process with the right arguments. Three layers do the work.

The provider layer owns keys and decides how a model is sealed and served.
The scheme layer holds the key-derived transforms and the sealed-checkpoint format.
The launch layer turns a description of a server into an engine command line and executes it.

Source layout

Path under `src/kllm/`	Responsibility
`cli.py`	The `kllm` command: argument parsing, passphrase input, and the order in which a command calls the other layers. It never handles key material itself.
`demo.py`	`kllm demo`: the two-stage demonstration with right and wrong keys.
`trust.py`	The `TrustProvider` interface between the CLI and `kllm.keyed`, the default key path, and `load_provider()`, which raises `TrustUnavailable` when the keyed extras are not installed.
`gpu.py`	GPU inventory from `nvidia-smi`, filtered by `CUDA_VISIBLE_DEVICES`, and the mapping of GPU names to GPU families.
`recipes.py`	The `Recipe` type, the built-in recipes and recipe matching.
`backends/launch.py`	`LaunchSpec`, the list of supported engines (`ENGINES`) and `exec_server()`.
`keyed/provider.py`	`Provider`, the implementation of `TrustProvider`: key generation and unlocking, dispatch to the right sealer, and the preparation of keyed and client-keyed launches.
`keyed/keys.py`	`ModelKey`, HKDF derivation, fingerprints and the passphrase-encrypted key file.
`keyed/perm.py`	Key-derived permutations (Fisher-Yates driven by a ChaCha20 keystream).
`keyed/schemes/`	The decoder-LLM scheme: `transforms.py` (the `orth` and `perm` transforms), `seal.py` (the sealer and the sealed-directory layout) and `base.py` (`SUPPORTED_BASES` and the packing of secret files).
`keyed/l1/`	Names shared by the schemes (`trusted.kllm`, environment variable names) and the chunked AES-256-GCM container of `trusted.kllm` (`blob.py`).
`keyed/h3/`	Sealing and serving of MiniMax H3 pipelines.
`keyed/client/`	Client-keyed mode: the export of the cloud part, the user-side LLM client and proxy, the client-keyed Z-Image sealer, the keyless latent server and reply encryption.
`keyed/d1/`	The keyed Z-Image scheme used by `kllm demo`.
`keyed/tee/keysource.py`	Key sources: the environment and plug-ins registered under the `kllm.keysource` entry-point group.
`keyed/attest.py`	The user-side attestation binding check (`python -m kllm.keyed.attest`).
`kernels/`	Optional INT8 kernels for the MiniMax H3 diffusion transformer and the `sitecustomize.py` hook that installs them in SGLang processes.

Dependencies follow the layers. cli.py, trust.py, gpu.py, recipes.py and backends/ need only the standard library, so plain serving works without the keyed extras. kllm.keyed needs cryptography (the keyed extra) and, for sealing and serving sealed models, torch, safetensors and numpy (the seal extra).

The provider layer

kllm.trust.TrustProvider is a typing.Protocol. Its methods are the complete set of operations the CLI needs:

Method	Used by
`keygen(path, passphrase)`	`kllm keygen`; returns the fingerprint.
`unlock(path, passphrase)`	Every command that takes `--key`; raises `ValueError` on a wrong passphrase.
`seal(src, dst, key, transform, client, lock)`	`kllm seal`; `lock` (default true) also locks the weights, and `--no-lock` passes false.
`prepare_serve(spec, key, attacker_base, key_source)`	`kllm serve --key`, `kllm serve --key-source cdh`, `kllm unlock` and the demo. Turns a plain `LaunchSpec` into a keyed one.
`prepare_client_serve(spec)`	`kllm serve --client`.
`attested_key()`	`kllm serve --key-source cdh`; delegates to the `cdh` key source.
`random_key()`, `image_server(...)`	`kllm demo`.

load_provider() imports kllm.keyed.provider.Provider directly; the provider is not a plug-in. Provider.seal chooses the sealer from the checkpoint:

no model_index.json: the decoder-LLM sealer in kllm.keyed.schemes.seal;
model_index.json of a MiniMax H3 pipeline: kllm.keyed.h3;
model_index.json with --client: the client-keyed Z-Image sealer (kllm.keyed.client.zimage), which accepts only Z-Image pipelines;
any other model_index.json: the keyed Z-Image sealer of kllm.keyed.d1.

Provider.prepare_serve reads the seal metadata, refuses a key whose fingerprint differs from the recorded one, refuses client-keyed and unknown schemes, and for the l1 scheme decrypts the permuted tokenizer into a 0700 directory under /dev/shm and records its path in spec.tokenizer. The sealed weights are passed to the engine as they are.

The scheme layer

A scheme defines how tensors are rewritten and how a sealed directory is laid out. The decoder-LLM scheme l1 is implemented by Transforms in kllm/keyed/schemes/transforms.py:

Shape.from_config reads the model dimensions from config.json.
Transforms derives every permutation, sign, scale and orthogonal basis from the key on demand, per layer and per expert.
Transforms.apply rewrites one canonical tensor by name; apply_fp8 does the same for an FP8 weight and its scale. Writers into the residual stream are mapped by the residual transform, readers by its inverse.
sealed_name gives the tensor’s name in the sealed checkpoint (experts move; every other name is unchanged).

kllm/keyed/schemes/seal.py validates the configuration (check_config), streams the safetensors shards through Transforms, writes the permuted tokenizer into trusted.kllm and adds a kllm section to config.json. The layout is described in Sealed checkpoint format.

Every transform is exact in exact arithmetic. The tests compare the logits of a float64 reference forward of the plain and the sealed model to 1e-9; see Adding a model family.

The launch layer

LaunchSpec in kllm/backends/launch.py is a plain description of one server: model, tensor-parallel size, host, port, recipe, extra engine arguments, environment, engine, and optionally a tokenizer path, a diffusion flag or a command of kLLM’s own. argv() renders the command line in a fixed order: the engine’s base command, host and port, the recipe’s arguments, then the user’s engine arguments, so that user arguments override the recipe. merged_env() combines the current environment with the recipe’s and the spec’s variables. exec_server() replaces the kllm process with the engine through os.execvpe. See Adding an engine.

The request path of `kllm serve`

For kllm serve ./Qwen3-8B-sealed --key, cli.cmd_serve performs these steps:

Detect the GPUs with gpu.detect().
With --engine sglang and without --no-recipe, look up a recipe with recipes.find(model, gpu_family, name).
Set the tensor-parallel size: --tp, else the recipe’s, else the number of visible GPUs (at least 1).
Build a LaunchSpec with the model, host, port, recipe and passthrough arguments.
Apply production mode, then choose the mode: --client calls prepare_client_serve; --key-source cdh calls attested_key and prepare_serve; --key unlocks the key file and calls prepare_serve.
With --int8, add the INT8 kernel environment for a MiniMax H3 pipeline.
Print the summary; with --dry-run, print spec.render() and exit.
Call launch.exec_server(spec).

From that point the engine serves requests on its own. kLLM adds no process, proxy or hook on the request path of an engine started by kllm serve; the only exception is the optional INT8 hook for MiniMax H3. For a client-keyed Z-Image pipeline, kllm serve starts kLLM’s own latent server (spec.command) instead of an engine.

Extension points

To add	Change
A decoder model family	`Transforms._layer_tensor`, `SUPPORTED_BASES`, `check_config` and `tests/test_families.py`. See Adding a model family.
An engine	`ENGINES` and `LaunchSpec.argv` in `backends/launch.py`, and the user-side engine client for client-keyed mode. See Adding an engine.
A serving recipe	`BUILTIN` in `recipes.py` and `tests/test_recipes.py`. See Contributing a serving recipe.
A key source	A separate package that registers an entry point in the `kllm.keysource` group. See Registering a key source.

The scope rule in CONTRIBUTING.md applies to all of them: a sealed checkpoint keeps every tensor’s name, shape and dtype so that the engine runs its stock model class and kernels. A change that needs engine patches or extra work per token is out of scope. See Contributing to kLLM.