Seal and serve a checkpoint by hand

This page seals Qwen3-1.7B under a new key with the lower-level commands of kLLM 0.1.0.dev0, serves it with SGLang or vLLM and sends a first request through the OpenAI-compatible API. It is the procedure behind the shorter workflow of the Quickstart, and it is the procedure to use for a checkpoint directory that is not in the model library. It needs one NVIDIA GPU and an installation with the seal extra and an engine extra (Installation guide).

Warning

This walkthrough uses kllm serve --key on an ordinary machine. Outside a confidential VM, --key decrypts the tokenizer into the host’s /dev/shm, where the host can read it; it offers no protection from the host. It is a development mode. For deployments without a confidential VM, use client-keyed mode (Client-keyed mode); for a confidential VM, see Confidential VM deployment.

Step 1: download a checkpoint

kllm seal reads a local Hugging Face checkpoint directory. Download one with the hf command of the huggingface_hub package:

$ hf download Qwen/Qwen3-1.7B --local-dir models/Qwen3-1.7B

Step 2: create a model key

$ kllm keygen

kllm keygen creates a 256-bit model key and writes it to ~/.kllm/keys/default.key, encrypted under a passphrase that it asks for twice. The passphrase must be at least 10 characters long. The command prints the key’s fingerprint.

  • -o PATH writes the key elsewhere; every keyed command then takes --key PATH.

  • --force overwrites an existing key file. Without it, kllm keygen refuses to overwrite.

Important

Back up the key file and its passphrase. Without both, a model sealed under the key cannot be served, and there is no recovery.

The passphrase protects the key file at rest only. The key itself is 32 random bytes, so a holder of the sealed files without the key file faces a 256-bit search for the key regardless of the passphrase; the transforms of a sealed public model are recoverable without that search, as stated at the end of this page. For unattended development starts, every command reads the passphrase from KLLM_PASSPHRASE when it is set; without a terminal and without that variable, the commands exit with an error. Key handling is described in Model keys.

Step 3: seal the checkpoint

$ kllm seal models/Qwen3-1.7B -o models/Qwen3-1.7B-sealed --transform perm

kllm seal asks for the key passphrase, rewrites the weights under the key and writes the sealed checkpoint to the output directory, which must not exist yet. --transform takes orth (the default) or perm; perm keeps every weight a bit-exact rearrangement of an original one. Choosing a transform compares the two. In the published measurements, sealing took 14 to 20 seconds for models of 1 to 2 billion parameters.

The sealed checkpoint keeps the architecture, tensor names, shapes and dtypes of the original. The weights are written to new safetensors shards, and a tied LM head is stored as a separate tensor. tokenizer.json and vocab.json are replaced by trusted.kllm, which holds the permuted tokenizer encrypted under the key, and config.json records the transform and the key fingerprint (Sealed checkpoint format).

Step 4: serve the sealed model

$ kllm serve models/Qwen3-1.7B-sealed --key --host 127.0.0.1

--key without a value uses ~/.kllm/keys/default.key; --key PATH names another file. kllm serve unlocks the key, checks its fingerprint against the sealed model, unlocks the locked weights into ~/.cache/kllm/unlocked on the first start (Locked weights), decrypts the tokenizer into memory and replaces itself with the engine process. The transform is read from the sealed model. The server listens on port 8000 by default (--port changes it) and prints the address of its OpenAI-compatible API.

--host defaults to 0.0.0.0; the command above binds the loopback interface only. To use vLLM instead of SGLang:

$ kllm serve models/Qwen3-1.7B-sealed --key --host 127.0.0.1 --engine vllm

Arguments after -- go to the engine unchanged, for example kllm serve models/Qwen3-1.7B-sealed --key -- --context-length 32768 for SGLang. Add --dry-run to print the engine command instead of running it.

Step 5: send a request

The served model is an ordinary OpenAI-compatible endpoint, and clients change nothing. List the served model first; by default both engines name it after the path given to kllm serve:

$ curl -s http://127.0.0.1:8000/v1/models

Then send a chat completion, with the identifier reported above as model:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "models/Qwen3-1.7B-sealed",
       "messages": [{"role": "user", "content": "Explain RoPE in two sentences."}],
       "max_tokens": 256, "temperature": 0}'

With the openai Python package (pip install openai):

from openai import OpenAI

# The engine is started without an API key; the client requires a placeholder value.
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
model = client.models.list().data[0].id

reply = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Explain RoPE in two sentences."}],
    max_tokens=256,
    temperature=0,
)
print(reply.choices[0].message.content)

The repository’s examples/client.py does the same with the standard library only:

$ python examples/client.py --port 8000 "Explain RoPE in two sentences."

Running the whole sequence as a script

examples/quickstart.sh performs steps 1 to 5 in one run. It writes the key to quickstart.key in the current directory, seals under perm, serves on port 8000 and calls examples/client.py. Set ENGINE=vllm to use vLLM.

$ examples/quickstart.sh
$ ENGINE=vllm examples/quickstart.sh

Warning

Unless KLLM_PASSPHRASE is already set, the script uses the fixed passphrase quickstart. Use it for evaluation only, and do not reuse its key for a model you intend to protect.

What the key protects in this setup

Without the key, the sealed directory is not usable through kLLM: the tokenizer cannot be decrypted, kllm serve refuses a key with the wrong fingerprint, and the weights produce no useful output with the public tokenizer. Qwen3-1.7B is a public model, however, and a sealed copy of a public model keeps neither its transforms nor its vocabulary permutation from a party that holds the public checkpoint: the measured attack recovers them from the sealed weights alone, without the key (Weights of a public model, or of a fine-tune of one). The key protects this sealed copy against parties that do not hold the public weights. In this development setup the host can also read the decrypted tokenizer in /dev/shm, and an operator who reads GPU memory sees user text as a substitution cipher over tokens. Deployment profiles and Claims state what each deployment profile protects.

Next steps