Quickstart

Note

Part of the upcoming 0.2 line. kllm pull, kllm run, kllm key and kllm create are present on the main branch and are not part of a tagged release. Install from a source checkout (Installation guide). Their options may change before 0.2.0; the command-line reference describes the current state. The lower-level procedure of kLLM 0.1.0.dev0 is described in Seal and serve a checkpoint by hand.

kLLM runs models from a curated library on production engines, SGLang and vLLM, and unlocks sealed models with a key. This page shows two workflows:

Both need one NVIDIA GPU with enough memory for the chosen model, and an installation with an engine extra (Installation guide). Check the host first:

$ kllm doctor

kllm doctor reports the Python version, the driver and GPUs, the CUDA version, the free disk space, the size of /dev/shm, the server port, the permissions of the kLLM home and the key directory, and the installed engine and download packages, each with its status (ok, warn or fail) and a suggested fix.

Run an official sealed model with its model key

The kLLM team seals selected public models ahead of time, each under its own key, and publishes them in the ANetResearch organization on Hugging Face, for example Qwen3-8B-kllm (The catalog of official sealed models). The catalog is rolling out; no catalog model has been confirmed as published yet. kllm search sealed lists the catalog models in the current library and their status. Their weights are locked: loaded into a stock engine without kLLM and the key, a catalog model computes nothing useful; with kLLM and its model key it runs normally.

Step 1: apply for model keys

Apply on the application page of the kLLM website. The application asks for your name, email address, organization, role, country, intended use and the catalog models you apply for; one application can cover several models. The kLLM team reviews each application. After approval, you receive a one-time, expiring download link to the key file of each granted model, for example Qwen3-8B-kllm.key, together with a statement of what a model key is and is not. Model keys describes model keys and their limits.

Step 2: import the key

$ kllm key import qwen3-8b-kllm Qwen3-8B-kllm.key
$ kllm key list

kllm key import checks the file’s format and copies it into the key ring, ~/.kllm/keys/qwen3-8b-kllm.key, with mode 0600. kllm key list prints the key’s name and fingerprint; no passphrase is needed for that.

Step 3: run the model

$ kllm run qwen3:8b-sealed

The library alias qwen3:8b-sealed names hf://ANetResearch/Qwen3-8B-kllm; either form works. Each catalog model has such an alias, its base’s tag with -sealed appended (Official sealed models).

kllm run downloads the model into the local store if it is not there yet, checks the download against the model’s published manifest, starts a background server on 127.0.0.1 and opens a chat session. For a sealed model it finds the key by fingerprint in the key ring, so --key is not needed, and asks for the key file’s passphrase on the terminal, or reads it from KLLM_PASSPHRASE. On the first start the server unlocks the weights into a cache directory, ~/.cache/kllm/unlocked by default, before the engine loads them; this takes extra time once and needs free disk space for one copy of the weights. Type /bye to end the session; /help lists the other session commands.

To answer one prompt and exit:

$ kllm run qwen3:8b-sealed "Explain RoPE in two sentences."

The download can be done separately with kllm pull qwen3:8b-sealed. kllm show qwen3:8b-sealed reports the base and revision the model was sealed from, whether its weights are locked, the fingerprint of its key and the matching key name. Model library and naming describes the library, the naming and the local store. To check the download against the public base, see Checking a published model.

Step 4: use the server

The background server keeps running after the session ends. It is an ordinary OpenAI-compatible endpoint on the loopback interface, on the first free port from 11400 upward:

$ kllm ps
$ curl -s http://127.0.0.1:11400/v1/models

Use the port that kllm ps reports. Any OpenAI-compatible client works without changes; see Seal and serve a checkpoint by hand for request examples. Stop the server when it is no longer needed:

$ kllm stop qwen3:8b-sealed

kllm run --rm stops the server automatically when the session ends, if that command started it.

Warning

Every applicant approved for a catalog model holds the same model key, and the catalog models are sealed copies of public models. Anyone who holds the key, or the public weights together with the unlocked weights, can read what this setup processes. Treat every prompt and output as readable by such a party.

Seal a model under a private key

A private key is a key that only you hold. kLLM generates it locally, and the sealing runs on your own machine with the open-source tool. The transform algorithms are open, so anyone can verify that a model is sealed as stated; what stays secret is the key.

Step 1: create a key

$ kllm key new prod

kllm key new creates a key with the existing key-file code, asks twice for a passphrase of at least 10 characters (or reads it from KLLM_PASSPHRASE), writes the encrypted key file to ~/.kllm/keys/prod.key and prints its fingerprint.

Important

Back up the key file and its passphrase. Without both, a model sealed under the key cannot be served, and there is no recovery.

Step 2: download and seal

$ kllm pull qwen3:8b
$ kllm create my-qwen3:8b --from qwen3:8b --key prod --transform perm

kllm create seals the plain model with the existing kllm seal command, which asks for the key passphrase, and stores the result as the local model my-qwen3:8b, with a manifest that records the model it was sealed from. The sealed model’s configuration records the transform and the key fingerprint. The weights are locked, as kllm seal locks them by default; kllm create --no-lock leaves them readable. The source model must be in the store first. --transform takes orth (the default where the library lists it) or perm; Choosing a transform compares the two. Sealing requires the seal extra and a supported architecture (Supported models). In the published measurements, sealing took 14 to 20 seconds for models of 1 to 2 billion parameters.

Step 3: run the sealed model

$ kllm show my-qwen3:8b
$ kllm run my-qwen3:8b

kllm show prints the transform, the key fingerprint and the matching key name. kllm run finds the key prod by its fingerprint and asks for its passphrase.

To serve a model under a private key on a machine operated by someone else, choose a deployment profile first (Deployment profiles). In client-keyed mode, which seals with kllm seal --client, the key and the tokenizer stay on your machine (Client-keyed mode).

What the key protects

  • Model key of a catalog model. Nothing confidential. Every applicant approved for the model holds the same key, and the models are public. It demonstrates the workflow.

  • Private key, run locally. kllm run unlocks the model on the machine it runs on, as kllm serve --key does: it decrypts the permuted tokenizer into /dev/shm, unlocks the weights into a cache directory and passes the passphrase to the server through its environment. It protects nothing from that machine, which is assumed to be yours. The sealed files themselves are useless without the key: the tokenizer is encrypted and the weights are locked (Locked weights).

  • Private key, public model. Once the weights are unlocked to run, a holder of the public checkpoint recovers the transforms and the vocabulary permutation from a sealed copy of a public model, and from a sealed fine-tune of one, without the key (Weights of a public model, or of a fine-tune of one). Weights with no public relative have no known attack under orth, and no proof either.

Claims states exactly what each deployment profile claims and does not claim.

Next steps