kLLM 0.1 documentation

Run any model. Protect what matters.

kLLM seals a model checkpoint under a 256-bit key and serves it on stock SGLang or vLLM. The sealed checkpoint keeps its tensor names, shapes and dtypes, so the engine runs it with its own model classes and kernels, while the tokenizer that maps its token ids to text is encrypted under the key and, by default, the weights on disk are encrypted in place until kLLM unlocks them with the key. kLLM is a layer on top of the inference engine, not a replacement for it. What sealing protects, and what it does not, depends on the deployment profile; see Claims.

The upcoming 0.2 line adds a workflow in the manner of Ollama, on production engines: kllm pull and kllm run with a curated model library that grows with adaptation work, a catalog of official sealed models unlocked with model keys issued on application, and private keys for models sealed on the user’s own machine (Quickstart).

This documentation covers kLLM 0.1.0.dev0 and, where marked, the commands of the upcoming 0.2 line on the main branch. It is organized as follows.

Getting help

Read the FAQ and the troubleshooting guide.
Look up a command in the command-line reference.
Report defects and ask questions in the issue tracker.

First steps

kLLM at a glance: Understand what kLLM does and where it sits relative to SGLang and vLLM.
Installation guide: Install kLLM, its engine extras and the container image.
Quickstart: Run an official sealed model with its model key, or seal one under a private key.
Examples: Run the end-to-end examples: keyed serving, client-keyed serving and MiniMax H3.

Basic concepts

Concepts: Get a map of model keys, sealed checkpoints, transforms and trusted domains.
Model library and naming: Refer to models by alias, and understand the model library and the local store.
Model keys: Create, protect and supply model keys.
The catalog of official sealed models: Find the official sealed models, run them and check them against their public base.
Model keys: Apply for the model keys of catalog models, and understand their limits.
Sealing: Choose a transform and seal a supported checkpoint.
Serving: Serve plain or sealed models on SGLang or vLLM, with recipes and engine arguments.
Client-keyed mode: Serve without a confidential VM while the key stays on the user’s machine.
Supported models: See the supported architectures, transforms and engines.
Requesting model support: Request support for a model on the public wishlist.
Deployment profiles: Understand the deployment profiles and what each one claims.
Sealed checkpoint format: Understand the layout of a sealed checkpoint.

Security

Threat model: Understand who the adversary is in each deployment profile.
Exposure: See what each party holds, and under which transform.
Attacks: Reproduce the weight-recovery and token-frequency attacks, and see what locked weights change.
Claims: Read exactly what kLLM claims and what it does not claim.

Deployment

Docker: Build and run the SGLang or vLLM container image.
Confidential VM deployment: Deploy in a confidential VM with a key broker.
Verifying a deployment: Check, as a user, that a deployment terminates TLS inside an attested VM.
NVIDIA Dynamo: Run sealed models under NVIDIA Dynamo.
Kubernetes and llm-d: Run sealed models on Kubernetes and llm-d.
The kLLM website: Build and deploy the kLLM website and the administration console.
The key-application service: Deploy and operate the key-application service.
Documentation hosting: Build and publish this documentation at docs.agentnetwork.org.cn.

Solving specific problems

Frequently asked questions: Get answers to the most frequently asked questions.
Seal and serve a checkpoint by hand: Seal and serve a checkpoint directory with the lower-level commands.
Choosing a transform: Decide between orth and perm for a given model.
Serving with vLLM: Serve with vLLM instead of SGLang.
Sealing FP8 and mixture-of-experts checkpoints: Seal FP8 and mixture-of-experts checkpoints.
Sealing and serving MiniMax H3: Seal and serve the MiniMax H3 FL2VA pipeline.
Running the H3 diffusion transformer on INT8 kernels: Run the H3 diffusion transformer on INT8 kernels.
Generating images in client-keyed mode: Generate images in client-keyed mode.
Benchmarking plain and sealed serving: Compare plain and sealed serving on other hardware.
Troubleshooting: Diagnose wrong keys, missing extras and engine errors.

Extending kLLM

Architecture overview: Understand the provider, scheme and launch layers.
Adding a model family: Add an architecture, with a float64 exactness test.
Adding an engine: Add a launch backend.
Contributing a serving recipe: Contribute a measured serving recipe.
Registering a key source: Register a key source through an entry point.

Reference

Command-line reference: Every command, option and default of the kllm tool.
Settings: Every environment variable kLLM reads or sets.
Glossary: Definitions of the terms used in this documentation.

Research

SCX: stateless KV-cache encoding: SCX: Stateless KV-Cache Encoding for Cloud-Scale Confidential Transformer Serving (ACM SIGCOMM 2025).
STIP: three-party privacy-preserving inference: STIP: Three-Party Privacy-Preserving and Lossless Inference for Large Transformers in Production (NDSS 2026).
Citing kLLM: Cite kLLM and its research, and find the research code and the official models.

All the rest

Release notes: See what changed in each release.
Roadmap: See the 0.2 line and what is planned: a local service, an Ollama-compatible API, a console, installers and confidential GPUs.
kLLM Enterprise: Learn about kLLM Enterprise: custom sealing, day-zero model support, deployment and attested key release.
Contributing to kLLM: Contribute code, model support and measurements.
Versioning and format stability: Understand version numbers and the stability of the sealed-checkpoint format.
License: Understand the Apache-2.0 core and the separately licensed Enterprise components.