kLLM at a glance

kLLM seals a model checkpoint under a 256-bit key and serves the sealed checkpoint on stock SGLang or vLLM. The tokenizer that turns the model’s output back into text is encrypted under the key and stays with the key holder. kLLM is an open-source project of Agent Network, released under the Apache-2.0 license.

kLLM is a layer on top of the inference engine, in the way LMCache is for KV caches. It does not replace SGLang or vLLM and does not fork them. The engine loads the sealed checkpoint with its own model classes and kernels, and kLLM adds key handling, sealing and launch logic around it.

Note

This documentation describes kLLM 0.1.0.dev0. The package is installed from source; see Installation guide.

Where kLLM sits

Seal once on the model owner’s machine; serve the sealed checkpoint on a stock engine.

A deployment has two phases.

Seal

On the model owner’s machine, kllm keygen creates a 256-bit model key, stored in a file encrypted with a passphrase (scrypt and AES-256-GCM). kllm seal rewrites a Hugging Face checkpoint under that key. The sealed checkpoint keeps the architecture, tensor names, shapes and dtypes of the original. Its tokenizer is permuted and stored encrypted in trusted.kllm; config.json records the transform and the key fingerprint.

Serve

kllm serve --key unlocks the key, decrypts the permuted tokenizer into memory and starts the engine on the sealed weights. Clients use the ordinary OpenAI-compatible API of the engine and send and receive text. The GPU receives permuted token ids and computes on transformed weights; the decrypted tokenizer is never sent to it.

The trusted domain that holds the key and the decrypted tokenizer is either a confidential VM or, in client-keyed mode, the user’s own machine (Client-keyed mode).

Three properties

Sealed checkpoints

The sealed checkpoint is useless without the key to a party that does not also hold the public model the weights derive from. The tokenizer is encrypted under a key derived from the model key, and the sealed weights produce no useful output with the public tokenizer. kllm serve refuses a key whose fingerprint does not match the one recorded at sealing time. A party that holds the public checkpoint of a sealed public model, or of the base of a fine-tune, recovers the transforms and the vocabulary permutation from the sealed weights without the key (Weights of a public model, or of a fine-tune of one). Two transforms are available: orth (a dense random orthogonal basis for the residual stream, the default) and perm (a random signed permutation, under which every sealed weight is a bit-exact rearrangement of an original one). See Sealing and Choosing a transform.

Stock engines

Because the transforms keep every tensor’s shape and every attention head group intact, the engine runs the sealed model with its own kernels, CUDA graphs, batching and tensor parallelism. The transforms are exact in exact arithmetic (float64 tests to 1e-9 on small random models); in bf16 they change only rounding and accumulation order. Measured on one NVIDIA A40 48 GB (SGLang 0.5.21 and vLLM 0.31.0, bf16, 1024 input and 128 output tokens, GPU shared with another tenant’s light job, single runs), sealed output throughput at concurrency 16 matched plain serving in every measured keyed configuration and was never more than 0.3% lower. Qwen3-1.7B under vLLM scored 0.5560, 0.5562 and 0.5548 on MMLU (plain, perm, orth), within one standard error.

Explicit security model

kLLM states what each deployment profile protects and what it does not. Sealing does not encrypt GPU memory. Against an operator who reads GPU memory, user text reduces to a substitution cipher over tokens, and the transforms and vocabulary permutation of a public model, or of a fine-tune of one, can be recovered from the sealed weights by a party that holds the public checkpoint. These limits are measured and documented in Threat model and Claims.

Measured output throughput at concurrency 16 (tok/s), one A40, conditions above

Model	Engine	Plain	perm	orth
Qwen3-1.7B	vLLM 0.31.0	1097	1094	1095
Qwen3-8B	SGLang 0.5.21	333	333	332

Latency deltas are visible in some rows: for example, the median time to first token of Qwen3-1.7B under SGLang at concurrency 1 moved from 98.0 ms to 105.2 ms under perm. The full tables, including perplexity and log-probability deviations, are in Benchmarking plain and sealed serving.

Deployment profiles

Profiles are trust boundaries, not scores. Each one is described in Deployment profiles.

Profile	Implemented as	Claims	Does not claim
Private Seal	kllm keygen and kllm seal, served with --client without a TEE; the key and tokenizer stay on the user’s machine	The sealed checkpoint is useless without the key to a party that does not also hold the public model the weights derive from; cloud logs and relayed payloads hold token ids under the vocabulary permutation (LLM) or ciphertext (Z-Image results)	Protection against a cloud that reads GPU memory or analyses the weights; for a public model, or a fine-tune of one, that analysis recovers the transforms and the vocabulary permutation
Attested CPU	kllm serve --key-source cdh --production in a confidential VM (Intel TDX, AMD SEV-SNP, Hygon CSV); the key source and the gateway are kLLM Enterprise	The key and the decrypted tokenizer stay in confidential-VM memory; users can check that TLS terminates in the attested VM	GPU memory protection (the GPU is an ordinary passthrough device); not yet validated on TEE hardware

Note

The Official catalog profile (official sealed models of the catalog, run with model keys that every applicant approved for a model shares, with no confidentiality) is part of the upcoming 0.2 line and is rolling out with the catalog (Model keys). A Confidential GPU profile is planned. See Roadmap.

Important

kllm serve --key outside a confidential VM decrypts the tokenizer into the host’s /dev/shm and unlocks the weights into a cache directory on the host. It then offers no protection from the host and is a development mode.

Supported models

Family	Transforms	Notes
Qwen3, Qwen3-MoE	orth, perm	Experts are permuted; each expert’s MLP has its own transforms
Qwen2, Qwen2.5	orth, perm	Q/K/V biases follow their heads
Llama 3.x	orth, perm	Llama 3 RoPE scaling
FP8 checkpoints of the above	perm	Block-wise or per-tensor scales
MiniMax H3 (FL2VA)	perm (recommended), orth	Text and images in, video with audio out; served with SGLang
Z-Image	Own scheme	Client-keyed mode (Generating images in client-keyed mode)

Plain, unsealed models can be served with any architecture the chosen engine supports. Supported models has the architecture names and engine coverage.

What kLLM is not

• It does not compute on ciphertext. The GPU computes on transformed weights and permuted token ids, not on ciphertext.

• It does not protect GPU memory. A confidential VM protects CPU memory only; the GPU is an ordinary passthrough device.

• It is not a fork of SGLang or vLLM, and it does not replace them or the orchestrators that start them.

• It does not build TEE infrastructure. In a confidential VM it expects an existing measured guest image, Confidential Containers guest components and a key broker run by the deployer (Confidential VM deployment).

• It does not seal arbitrary models. Sealing supports the families listed above.

• In 0.1.0.dev0, it does not download models or keep a model library.

Note

Part of the upcoming 0.2 line. kllm pull, kllm run, a curated model library and a named key ring are on the main branch and are not part of a tagged release. See Quickstart and Model library and naming.

Research

The kLLM team publishes research on confidential transformer serving. The perm transform extends STIP’s permutation approach to grouped-query attention, RoPE, MoE and FP8; the research systems themselves are separate from what kllm seal ships today. See SCX: stateless KV-cache encoding and STIP: three-party privacy-preserving inference.

Licensing

kLLM is Apache-2.0. The confidential-VM key sources (cdh, gramine) and the attesting TLS gateway are part of kLLM Enterprise, licensed separately. The user-side binding check, python -m kllm.keyed.attest, is open source. See License.

Next steps

• Installation guide describes the requirements and the installation from source.

• Quickstart seals and serves a first model.

• Examples walks through keyed serving, client-keyed serving and MiniMax H3.