Frequently asked questions

This page answers the questions most often asked about kLLM 0.1.0.dev0. Each answer states what the current release does; planned features are marked as such. For error messages, see Troubleshooting.

Is sealing a form of encryption?

Not of the weights. kllm seal rewrites the checkpoint with static transforms derived from a 256-bit model key: a dense random orthogonal basis (orth) or a signed permutation (perm) for the residual stream, permutations of attention heads, of the MLP’s intermediate dimension and of the regular vocabulary. The network computes the same function in a secret basis. The only encrypted artifact is the permuted tokenizer, stored in trusted.kllm under AES-GCM with a key derived from the model key.

The GPU therefore computes on transformed weights and permuted token ids, not on ciphertext. Against an operator who reads GPU memory, user text reduces to a substitution cipher over tokens. See Sealing and Claims.

Does sealing slow down inference?

Served with a key, sealed throughput matched plain serving in every measured configuration on one NVIDIA A40 (never more than 0.3% lower output throughput at concurrency 16). For example, Qwen3-1.7B under vLLM 0.31.0 produced 1097 tok/s plain, 1094 tok/s under perm and 1095 tok/s under orth; Qwen3-8B under SGLang 0.5.21 produced 333, 333 and 332 tok/s.

Latency differences are visible in some rows: median time to first token of Qwen3-1.7B on SGLang at concurrency 1 was 98.0 ms plain and 105.2 ms under perm, and median time per output token of Qwen3-8B at concurrency 16 was 35.3 ms plain and 37.3 ms under orth. Client-keyed serving under vLLM cost 7% throughput at concurrency 16 (1036 to 963 tok/s). All figures are single runs on a GPU shared with another tenant’s light job; see Benchmarking plain and sealed serving for the conditions and the full tables.

Does sealing change the model’s outputs?

The transforms are exact in exact arithmetic; the test suite checks this in float64 to 1e-9 on small random models. Under perm every sealed weight is an original value, reordered and possibly sign-flipped. In bf16 serving only the order of floating-point accumulation changes: the mean absolute log-probability difference against plain serving was 0.03 to 0.10 for the measured Qwen3 models, against 0.053 for switching plain Qwen3-1.7B from bf16 to fp16.

On standard evaluations, Qwen3-1.7B under vLLM scored MMLU 0.5560 / 0.5562 / 0.5548 and GSM8K (strict) 0.6945 / 0.6952 / 0.6960 (plain / perm / orth), all within one standard error.

Which models can be sealed?

The architectures Qwen3ForCausalLM, Qwen3MoeForCausalLM, Qwen2ForCausalLM and LlamaForCausalLM (Qwen3, Qwen3-MoE, Qwen2 and Qwen2.5, Llama 3.x), FP8 checkpoints of these families with block-wise or per-tensor scales (perm only), the MiniMax H3 FL2VA pipeline, and Z-Image in client-keyed mode. kllm seal rejects other architectures with an error that lists the supported ones. Plain serving, without a key, covers every model the chosen engine supports. See Supported models.

Which inference engine should be used?

SGLang is the default (--engine sglang); vLLM is selected with --engine vllm. Both run sealed LLMs under orth and perm, FP8 checkpoints and client-keyed mode. Only SGLang serves MiniMax H3, runs the INT8 DiT kernels and uses the built-in recipes. See Serving with vLLM for the differences.

Does kLLM fork or patch SGLang or vLLM?

No. A sealed checkpoint keeps its file layout, tensor names, shapes and dtypes, so the engine loads it with its own model class and runs its own kernels, CUDA graphs, batching and tensor parallelism. kllm serve builds the engine’s command line and replaces itself with the engine process. The INT8 DiT kernels for MiniMax H3 are the one exception: when requested with --int8, they replace SGLang’s attention and linear implementations inside the H3 DiT blocks at run time (see Running the H3 diffusion transformer on INT8 kernels).

What does the cloud see?

It depends on the deployment profile, which is a trust boundary rather than a score:

• Private Seal (kllm seal --client and kllm serve --client, no TEE): the key and the tokenizer stay on the user’s machine. The cloud’s logs and relayed payloads hold token ids under the vocabulary permutation, or ciphertext for Z-Image results. A cloud that reads GPU memory or analyses the weights is not defended against; the cloud’s copy of the weights is unlocked, and for a public model, or a fine-tune of one, analysing it against the public checkpoint yields the vocabulary permutation and with it the text behind the token ids.

• Attested CPU (kllm serve --key-source cdh --production in a confidential VM): the key and the decrypted tokenizer stay in confidential-VM memory. GPU memory is not protected, and this profile has not yet been validated on TEE hardware.

In both profiles, an operator who reads GPU memory sees user text as a substitution cipher over tokens, and a serving process that is attached to or modified sees everything it computes. See Deployment profiles and Exposure.

Is kllm serve --key safe on an ordinary cloud host?

No. Outside a confidential VM, kllm serve --key unlocks the key on the host, decrypts the tokenizer into host /dev/shm and unlocks the weights into a cache directory on the host, so it offers no protection from that host. It is a development mode. To keep the key off the cloud host, use client-keyed serving (Client-keyed mode).

Can someone with the sealed files but without the key run the model?

Not usefully. kllm serve refuses a key whose fingerprint does not match the sealed model, and the tokenizer cannot be decrypted without the right key. By default the weights are also locked, encrypted in place, so they compute nothing useful in any engine and cannot be matched to the public base without the key (Locked weights). For models sealed with --no-lock, run with the public tokenizer, a sealed Qwen3 model emits unrelated tokens, and sealed MiniMax H3 produces an unrelated video; a holder of such files who also has the public model from which they derive, whether an unmodified public model or the base of a fine-tune, recovers the transforms and the vocabulary permutation from the sealed weights without the key. The same holds for anyone who obtains the weights unlocked, for example the cloud in client-keyed mode or the cache of a server that runs the model. See the next question.

Does sealing protect the weights of a fine-tuned model?

Not against a party that has the public base model and analyses the sealed weights once they are unlocked to run; locking protects only the files on disk. A fine-tune of a public base is recoverable from the unlocked sealed weights under both transforms: under orth the measured attack succeeds for weight changes up to 3% of the weight standard deviation, and under perm at any fine-tune size. The same attack applies to a sealed copy of an unmodified public model, which is the eps = 0 case of the measurement under perm: it recovers the transforms and 98.5% of the vocabulary permutation on Qwen3-8B, the remainder being unused tokens. Sealing a public model therefore hides neither the model nor the vocabulary permutation from anyone who has the public weights and the unlocked sealed weights, whatever key is used. For weights with no public relative there is no known attack under orth, and no proof of security either. See Attacks.

How is the key of an official sealed model obtained?

Apply for model keys on the application page. Each catalog model has its own key, and one application can cover several models. Every applicant approved for a model receives the same key for it, so a model key demonstrates the workflow and is not a confidentiality boundary. See Model keys and The catalog of official sealed models.

How can an official sealed model be checked against its public base?

With the model key, kllm verify MODEL --base REPO --key KEYFILE seals the public base again under the same key and compares the result with the published model tensor by tensor. Under perm every tensor must be equal to the bit. The sealing code is open source, so a successful check shows that the published files are what that code makes of the public base. kllm check only re-hashes local files against their manifest. See Checking a published model.

Should orth or perm be used?

perm is required for FP8 checkpoints and recommended for MiniMax H3; it keeps every weight bit-exact and showed the smaller log-probability drift. orth (the default) mixes coordinates, so that the per-channel statistics that survive under perm are not directly exposed, at the cost of rounding the rotated weights again. Neither transform prevents the attacks on token ids and on fine-tunes of public models described in Attacks. Choosing a transform compares them in detail.

What happens if the key file or its passphrase is lost?

The sealed model can no longer be served. kLLM keeps no copy of the key and has no recovery mechanism. Back up the key file together with its passphrase, or seal the original checkpoint again under a new key. See Model keys.

What is kLLM Enterprise?

kLLM Enterprise is the separately licensed set of components for production deployment in confidential VMs: the confidential-VM key sources (cdh, gramine), which obtain the model key from the deployer’s key broker after attestation, and the attesting TLS gateway. The rest of kLLM, including the user-side binding check python -m kllm.keyed.attest, is open source under Apache-2.0. With the open-source package alone, kllm serve --key-source cdh exits with key source 'cdh' needs kLLM Enterprise (kllm-enterprise). See License and Confidential VM deployment.

How does kLLM relate to Ollama?

They address different needs. Ollama offers a one-command pull and run of plain models for local, single-user use. kLLM seals models under a key and serves them on SGLang or vLLM, typically on rented GPUs. The sealing and serving commands of 0.1.0.dev0 include no pull command, no model aliases and no interactive run command.

The upcoming 0.2 line brings the Ollama workflow to production engines: kllm pull and kllm run with model aliases such as qwen3:8b, a curated model library, and a catalog of official sealed models unlocked with model keys (Quickstart). These commands are on the main branch and are not part of a tagged release. A resident local service with an Ollama-compatible API is planned; see Roadmap.

How does kLLM relate to SCX and STIP?

The kLLM team publishes research on confidential transformer serving. The perm transform extends STIP’s permutation approach to grouped-query attention, RoPE, MoE and FP8; the research systems themselves are separate from what kllm seal ships today. kllm seal does not provide the security properties of SCX or STIP. See SCX: stateless KV-cache encoding, STIP: three-party privacy-preserving inference and Citing kLLM.

Does kLLM work with Dynamo, llm-d and Kubernetes?

Orchestrators start the engine themselves. kllm unlock MODEL decrypts a sealed model’s tokenizer into memory and prints its path for the engine’s tokenizer argument; for a model with locked weights it also unlocks the weights into a cache directory, which the engine then loads as the model (Other launchers: kllm unlock). Examples for NVIDIA Dynamo, llm-d and Kubernetes are included in the repository but have not been run end to end. A component that tokenizes, such as a Dynamo frontend, holds the secret vocabulary and belongs in the trusted domain (Trusted domain). See NVIDIA Dynamo and Kubernetes and llm-d.

Are confidential GPUs supported?

No. The serving GPU is an ordinary device whose memory the cloud can read.

Note

Planned — not available in 0.1.0.dev0. Confidential-GPU support, and validation of the Attested CPU profile on TDX, SEV-SNP and Hygon CSV hardware. See Roadmap.