Confidential VM deployment

This page describes the Attested CPU deployment profile: the serving stack runs inside a confidential VM, the model key is released to that VM by the deployer’s key broker after attestation, and users check that TLS terminates inside the attested VM before they send data. The profile and its claims are defined in Deployment profiles; what each party holds is listed in Exposure.

Important

The cdh and gramine key sources and the attesting TLS gateway are part of kLLM Enterprise (kllm-enterprise), which is licensed separately. With the open-source package alone, kllm serve --key-source cdh exits with key source 'cdh' needs kLLM Enterprise (kllm-enterprise). The user-side binding check, python -m kllm.keyed.attest, is part of open-source kLLM and is described in Verifying a deployment.

Warning

This deployment has not yet been validated on Intel TDX, AMD SEV-SNP or Hygon CSV hardware. The GPU is an ordinary passthrough device: its memory, and the PCIe traffic to it, are readable by the cloud. The profile does not protect GPU memory; see Threat model.

Note

Planned — not available. Validation of this deployment on TDX, SEV-SNP and Hygon CSV hardware. See Roadmap.

Components

kLLM does not build trusted-execution infrastructure. It relies on the following components, which the deployer provides and operates:

Component	Provided by	Role
Confidential VM	The deployer, on a TDX, SEV-SNP or Hygon CSV host	Runs kllm serve, the engine and the gateway. Its CPU and memory are not readable by the cloud. An ordinary GPU is passed through to it.
Measured guest image	The deployer, from an existing measured image	Defines the measurements that the key broker and the users check.
Attestation Agent and Confidential Data Hub	Confidential Containers guest-components	The Attestation Agent produces evidence. The Confidential Data Hub (CDH) attests the VM to the key broker and serves released secrets through its REST API (http://127.0.0.1:8006 by default).
Key broker (KBS)	Trustee, run by the deployer	Holds the model key as a resource and releases it only to VMs whose measurements match the deployer’s policy.
cdh key source	kLLM Enterprise	Lets kllm serve --key-source cdh obtain the model key from the CDH.
Attesting TLS gateway	kLLM Enterprise	Terminates TLS inside the VM and serves /attestation.
Binding check	Open-source kLLM	python -m kllm.keyed.attest, run by users (Verifying a deployment).

Prerequisites

• A confidential VM (Intel TDX, AMD SEV-SNP or Hygon CSV) with an ordinary GPU passed through. Build the guest from an existing measured image, for example Canonical’s TDX guest or the Confidential Containers guest, with dm-verity on the root file system and a reproducible build, so that the deployer and the users can both check its measurements. GPU DMA passes through swiotlb bounce buffers in shared memory; that traffic is visible to the cloud.

• Confidential Containers guest-components in the VM: the Attestation Agent and the Confidential Data Hub with its REST API server.

• A Trustee key broker (KBS), run by the deployer, holding the model key as a resource. The default resource path is default/kllm/model-key. The resource holds 32 raw bytes or 64 hexadecimal digits.

• kLLM with the seal extra and kLLM Enterprise installed in the VM, together with the engine (serve or vllm extra).

Procedure

1. Seal the model on the deployer’s machine

$ kllm keygen
$ kllm seal ~/models/Qwen3-8B -o Qwen3-8B-sealed --transform orth

Either transform can be used; see Choosing a transform. Upload the sealed directory to the cloud. The sealed directory contains no usable tokenizer: the permuted tokenizer is encrypted under the key in trusted.kllm. Its weights are locked, so the cloud’s storage holds nothing that can be matched to a public base without the key (Locked weights).

2. Store the key in the key broker

Store the model key in the KBS at the resource path that the VM requests (default/kllm/model-key unless overridden), as 32 raw bytes or 64 hexadecimal digits, behind a policy that releases it only to VMs with the expected measurements.

Note

kLLM 0.1.0.dev0 has no command that writes the raw model key. kllm keygen produces a passphrase-protected key file only, so provisioning the raw key into the broker is outside the open-source command-line tool.

3. Start the server in the VM

$ export KLLM_PRODUCTION=1
$ kllm serve /models/Qwen3-8B-sealed --key-source cdh --host 127.0.0.1 --port 8000

--key-source cdh asks the CDH for the key; the CDH first attests the VM to the KBS. On success kllm serve prints the fingerprint of the released key, checks that the sealed model was sealed with a key of that fingerprint, unlocks the weights into its cache directory, and decrypts the permuted tokenizer into a fresh directory with mode 0700 under /dev/shm, which is guest memory. For MiniMax H3 the directory is a view of the sealed pipeline with the tokenizer files in place. The engine then starts on the unlocked weights with that tokenizer, bound to the loopback interface so that only the gateway reaches it.

The cache of unlocked weights is ~/.cache/kllm/unlocked unless KLLM_UNLOCKED_DIR names another directory (Locked weights). It is written to the VM’s file system as ordinary files and needs room for one copy of the weights. Place it on storage that the cloud cannot read, such as a volume encrypted inside the VM or guest memory; on storage the cloud can read, the cloud obtains the unlocked weights from it.

The Enterprise environment variables KLLM_CDH_URL and KLLM_CDH_RESOURCE override the CDH address and the resource path.

4. Start the attesting gateway (kLLM Enterprise)

$ python -m kllm_enterprise.gateway --self-signed api.example.com \
    --model /models/Qwen3-8B-sealed --upstream http://127.0.0.1:8000 --port 8443

The gateway terminates TLS inside the VM, forwards requests to the engine and serves /attestation. With --self-signed the TLS key is generated in VM memory and never written to disk. Evidence comes from the Attestation Agent (KLLM_AA_URL, for example http://127.0.0.1:8006) or from the kernel’s configfs-tsm interface. The report data in the evidence binds the gateway’s TLS key:

report_data = hex(SHA-256("kllm-tls-v1" || SubjectPublicKeyInfo of the TLS key || nonce))

The full options of the gateway are documented with kLLM Enterprise.

5. Publish reference values

Publish the measurements of the guest image so that users can compare the quote against them. Users then follow Verifying a deployment before sending requests.

Production mode

--production, or KLLM_PRODUCTION set to any value other than empty or 0, makes kLLM accept the model key from the CDH only. kllm serve --production also sets KLLM_PRODUCTION=1 for the engine processes. In production mode kLLM refuses the following:

Request	Result
kllm serve SEALED --key [KEYFILE]	Exits with --production takes the model key from the CDH only (--key-source cdh).
A key in KLLM_MODEL_KEY	Refused by the key-source loader (kllm.keyed.tee.keysource.load_key): KLLM_PRODUCTION=1 refuses keys from the environment; use cdh.
kllm serve SEALED --client	Exits with --client serves without a key; the key stays with the user.
kllm unlock SEALED	Refused after the key file is unlocked: KLLM_PRODUCTION=1 takes the model key from the CDH only. Launchers such as Dynamo and llm-d therefore have no open-source production path in 0.1.0.dev0.

The --key-source option accepts passphrase and cdh. Other key sources, such as the Enterprise gramine source, are registered under the kllm.keysource entry-point group. The key-source loader selects the source named by KLLM_KEY_SOURCE (default env, which reads KLLM_MODEL_KEY); see Registering a key source.

Running without hardware

Every step above also runs on an ordinary machine, where it attests nothing: the gateway’s /attestation reports "source": "none", and KLLM_PRODUCTION=1 refuses to start. The algorithms and the traffic visible on the GPU are the same as in a confidential VM, so the tests, the benchmarks and the attack tools in tools/attacks/ apply unchanged.

Limitations

• GPU memory and PCIe traffic are visible to the cloud. Against an operator who reads GPU memory, user text reduces to a substitution cipher over tokens, and for a public model, or a fine-tune of one, the operator can recover the vocabulary permutation from the sealed weights; see Attacks and Claims.

• MiniMax H3 decoded video and audio are in GPU memory in the clear.

• The decrypted tokenizer directory in /dev/shm is not removed when the server exits. It remains in guest memory until it is deleted or the VM is restarted. The cache of unlocked weights is not removed either.

• A serving process that is attached to or modified sees everything it computes.

• The deployment has not been validated on TEE hardware.

See also

• Verifying a deployment for the user-side check.

• Model keys for key files, fingerprints and key handling.

• Settings for the environment variables named on this page.