Command-line reference

This page is the reference for the kllm command. Its first part lists every subcommand, positional argument and option of the core commands of kLLM 0.1.0.dev0 on the main branch, with its default and its help text exactly as defined in src/kllm/cli.py and src/kllm/demo.py. The section Model, key and runtime commands (0.2 line) describes the commands added in the upcoming 0.2 line. Task-oriented descriptions are in Quickstart, Model keys, Sealing, Serving and Client-keyed mode.

General behavior

The console script kllm and python -m kllm run the same entry point, kllm.cli:main. The top-level parser has the description keyed LLM inference.

kllm [-h] {version,recipes,keygen,seal,serve,unlock,verify,cloud,client,demo,pull,run,
           list,ls,ps,stop,rm,show,check,create,key,doctor,request,search,logs,
           completion} ...
-h, --help

Print the help of kllm or of any subcommand and exit. Every subcommand accepts it.

The core subcommands are version, recipes, keygen, seal, serve, unlock, verify, cloud, client and demo. verify, cloud and seal --no-lock were added on the main branch after the last published notes and are not part of a tagged release.

The upcoming 0.2 line adds pull, run, list (alias ls), ps, stop, rm, show, check, create, key, doctor, search, request, logs and completion; see Model, key and runtime commands (0.2 line). These commands resolve model aliases such as qwen3:8b. The existing commands keep their behavior: a model argument of kllm serve is passed to the engine as given.

Rules that apply to every subcommand:

  • Engine passthrough. Everything after the first -- on the command line is passed to the engine unchanged. Only kllm serve accepts it; any other subcommand followed by -- exits with `--` passthrough is only valid for `kllm serve`.

  • Passphrase. Every command that unlocks a key file reads the passphrase from KLLM_PASSPHRASE when it is set and non-empty, and otherwise prompts on the terminal. Without a terminal it exits with no terminal to read the key passphrase; set KLLM_PASSPHRASE. See Settings.

  • Default key path. Options named --key default to ~/.kllm/keys/default.key ($HOME/.kllm/keys/default.key, printed expanded in the help output). The options of the existing commands address keys by path; the 0.2-line commands also accept key names (kllm key).

  • Option abbreviations. The parser accepts unambiguous prefixes of long options, for example --trans for --transform. This documentation and scripts should use the full names.

Exit status:

Status

Meaning

0

Success. kllm serve does not return on success: it replaces its own process with the engine (execvpe), so the exit status is the engine’s.

1

The command refused to proceed and printed a message, for example an existing output directory, a wrong passphrase or an incompatible combination of options.

2

The command line could not be parsed, the keyed extras are missing (the message names kllm[seal]), a sealed model does not match the key or the mode, or a recipe name is unknown. These messages go to standard error.

kllm version

Help text:

print the version

Synopsis:

kllm version

Prints kllm followed by the package version, for example kllm 0.1.0.dev0. The subcommand takes no options.

kllm recipes

Help text:

list built-in serving recipes

Synopsis:

kllm recipes

Prints one line per built-in recipe: its name, its tensor-parallel size (tp=), the GPU families it was measured on (gpus=) and a one-line summary. The subcommand takes no options. kLLM 0.1.0.dev0 ships three recipes, all for GLM-5.3 on H200 or B300 GPUs and all for SGLang; see Contributing a serving recipe.

kllm keygen

Help text:

create a model key

Synopsis:

kllm keygen [-o OUTPUT] [--force]

Creates a 256-bit model key, prompts twice for a new passphrase (or takes KLLM_PASSPHRASE), writes the passphrase-encrypted key file with mode 0600 and prints its path and fingerprint. The passphrase must be at least 10 characters long. See Model keys.

-o OUTPUT, --output OUTPUT
Default:

~/.kllm/keys/default.key

Help text:

none

Path of the key file to write. Missing parent directories are created; the directory that directly contains the key file is created with mode 0700.

--force
Default:

off

Help text:

none

Overwrite an existing key file. Without it, an existing file stops the command with <path> exists; pass --force to overwrite.

Warning

Overwriting a key file destroys the key unless a copy exists elsewhere. Models sealed under that key can then no longer be served.

kllm seal

Help text:

seal a checkpoint so that it only runs under a key

Synopsis:

kllm seal MODEL -o OUTPUT [--key KEYFILE] [--transform {orth,perm}] [--client] [--cloud DIR] [--no-lock]

Writes a sealed copy of a local checkpoint. The sealer is chosen from the checkpoint: decoder LLMs (Qwen3, Qwen3-MoE, Qwen2/2.5, Llama 3.x, including FP8 checkpoints of these under perm), MiniMax H3 pipelines and Z-Image pipelines. Unless --no-lock is given, the command then locks the weights: it encrypts every weight in place under a key derived from the model key (Locked weights), and its success message ends with (weights locked). On success the command prints the matching kllm serve command. See Sealing and Supported models.

MODEL
Help text:

local Hugging Face checkpoint directory

A directory on the local file system. Hugging Face Hub identifiers are not accepted. A decoder LLM needs config.json, *.safetensors weights and a tokenizer.json.

-o OUTPUT, --output OUTPUT
Default:

required

Help text:

directory for the sealed model

The directory must not exist; the command exits with <OUTPUT> exists otherwise.

--key KEYFILE
Default:

~/.kllm/keys/default.key

Help text:

none

Key file to seal under. The command unlocks it before sealing.

--transform {orth,perm}
Default:

orth

Help text:

orth: dense orthogonal bases, which also hide coordinates and outliers; equal to the original up to rounding the rewritten weights. perm: signed permutations; every weight is a bit-exact rearrangement of an original one

FP8 checkpoints require perm. For MiniMax H3, perm is recommended. A Z-Image pipeline sealed without --client uses its own scheme and ignores this option. See Choosing a transform.

--client
Default:

off

Help text:

for serving without the key in the cloud (no TEE): also write the cloud’s part, without tokenizer or key material, to --cloud

Seals for client-keyed serving. For LLMs the sealed directory is the same as without the option; the command additionally exports the files the cloud needs. Pipelines with a model_index.json are accepted only if they are Z-Image pipelines. The command prints the cloud-side kllm serve command and the user-side kllm client proxy or kllm client generate command. See Client-keyed mode.

--cloud DIR
Default:

OUTPUT-cloud

Help text:

where --client puts the cloud’s part (default OUTPUT-cloud)

Used only with --client. The directory must not exist. When the weights are locked, the command writes the weight files of this directory unlocked, with the key it sealed under; the key itself is not copied. Other files, and every file when --no-lock is given, are hard-linked from OUTPUT where possible, so editing them in place also edits the sealed copy.

--no-lock
Default:

off

Help text:

leave the sealed weights readable. By default every weight is also encrypted in place: the files load in any engine but compute nothing until kllm serve --key unlocks them

Skips locking. The sealed weights are then readable on disk, and a party that holds them and the public model they derive from can recover the transforms and the vocabulary permutation without the key (Weights of a public model, or of a fine-tune of one). Use it only where every party that can read the files may also run the model, for example when the sealed directory is itself the cloud’s copy.

kllm serve

Help text:

serve a model; arguments after -- go to the engine unchanged

Synopsis:

kllm serve MODEL [--engine {sglang,vllm}] [--tp TP] [--host HOST] [--port PORT] [--key [KEYFILE]] [--key-source {passphrase,cdh}] [--production] [--int8 [PARTS]] [--client] [--recipe RECIPE] [--no-recipe] [--dry-run] [-- ENGINE_ARGS ...]

Detects the local GPUs, chooses a recipe, prepares the engine command and replaces itself with the engine process. It serves a plain model, a sealed model with its key (--key), a sealed model with a key released by a key source (--key-source), or a client-keyed cloud directory without a key (--client). Before starting the engine it prints the kLLM version, the engine, the GPU inventory, the recipe and the serving address. See Serving.

MODEL
Help text:

model path or Hugging Face id

Passed to the engine as given (--model-path for SGLang, the model argument of vllm serve). A sealed model must be a local directory.

--engine {sglang,vllm}
Default:

sglang

Help text:

inference engine to run

SGLang starts as python -m sglang.launch_server with the current interpreter; vLLM starts as the vllm console script installed next to it (vllm serve). MiniMax H3 pipelines are served with SGLang only.

--tp TP
Default:

the recipe’s size, otherwise the number of visible GPUs (at least 1)

Help text:

tensor parallel size

Passed as --tp (SGLang), --tensor-parallel-size (vLLM) or --num-gpus (sglang serve for MiniMax H3). Visible GPUs are those reported by nvidia-smi and selected by a numeric CUDA_VISIBLE_DEVICES.

--host HOST
Default:

0.0.0.0

Help text:

none

Address the server binds to.

Warning

The default binds every interface. A server started with --client has no authentication; bind 127.0.0.1 or place it behind a separate gateway.

--port PORT
Default:

8000

Help text:

none

Port the server listens on.

--key [KEYFILE]
Default:

absent; a bare --key means ~/.kllm/keys/default.key

Help text:

serve with a model key (default $HOME/.kllm/keys/default.key, printed as an absolute path)

Unlocks the key file with its passphrase and checks that the model was sealed under this key (by fingerprint). For a model with locked weights, it then writes an unlocked copy into the cache directory (Locked weights), once per key and set of weight files, and passes that copy to the engine as the model. It decrypts the permuted tokenizer into a new 0700 directory under /dev/shm and passes that directory to the engine (--tokenizer-path or --tokenizer). The key itself is not passed to the engine.

Because the value is optional, the model must come first: write kllm serve MODEL --key or kllm serve MODEL --key KEYFILE. In kllm serve --key MODEL the model path is taken as the key file.

Important

Outside a confidential VM, /dev/shm, the cache of unlocked weights and the passphrase belong to the host, and this mode offers no protection from the host. It is a development mode. See Deployment profiles.

--key-source {passphrase,cdh}
Default:

passphrase

Help text:

passphrase: unlock KEYFILE (development); cdh: inside a confidential VM, have the Confidential Data Hub attest the VM to the deployer’s key broker and release the key

cdh does not need --key. It requires a key source registered under the name cdh, which kLLM Enterprise provides; with the open-source package alone the command exits with status 2 and key source 'cdh' needs kLLM Enterprise (kllm-enterprise). See Registering a key source and Confidential VM deployment.

--production
Default:

off

Help text:

refuse keys from passphrases or the environment (also KLLM_PRODUCTION=1)

Production mode. Any value of KLLM_PRODUCTION other than empty or 0 has the same effect, and the option sets KLLM_PRODUCTION=1 for the engine. In production mode the command starts only with --key-source cdh; otherwise it exits with --production takes the model key from the CDH only (--key-source cdh). This applies to plain models as well.

--int8 [PARTS]
Default:

absent; a bare --int8 means all

Help text:

MiniMax H3: run the DiT blocks on INT8 kernels (attention, linear or all; default all). For GPUs without FP8 tensor cores; needs comfy-kitchen for linear

Accepted only for a sealed MiniMax H3 pipeline served with a key (--key or --key-source cdh); otherwise the command exits with --int8 applies to sealed MiniMax H3 pipelines (served with --key). PARTS is attention, linear, all or a comma-separated list. The option sets KLLM_INT8 and prepends the kernel directory to PYTHONPATH for the engine.

Important

INT8 is a kernel optimization separate from sealing, with measured numerical drift. Only the attention part is verified to commute with perm. See Running the H3 diffusion transformer on INT8 kernels.

--client
Default:

off

Help text:

serve a sealed model without its key: token ids in and out (LLMs, engine started with --skip-tokenizer-init) or latents out (Z-Image); the user runs kllm client

Serves the cloud directory written by kllm seal --client. For an LLM, kLLM adds --skip-tokenizer-init to the engine arguments. For a client-keyed Z-Image pipeline, kLLM starts its own latent server (python -m kllm.keyed.client.server) instead of an engine, without a recipe. The option cannot be combined with --key, --key-source cdh or production mode (--client serves without a key; the key stays with the user). If the directory contains files that belong on the user’s side, the command prints a warning and continues. A directory with locked weights is refused with <MODEL> is locked; export the cloud's copy with `kllm cloud <MODEL> -o DIR --key KEYFILE` on your side; see kllm cloud.

--recipe RECIPE
Default:

none

Help text:

force a recipe by name

Applies the named recipe regardless of model and GPU. Recipes are consulted only with --engine sglang and without --no-recipe; in other cases the option has no effect. An unknown name exits with status 2 and a message that refers to kllm recipes.

--no-recipe
Default:

off

Help text:

use the engine’s defaults

--dry-run
Default:

off

Help text:

print the launch command

Prints the engine command line, prefixed by the environment variables kLLM sets for the engine, and exits with status 0. KLLM_MODEL_KEY is shown as <secret>.

Note

With --key, the key is unlocked and the tokenizer is decrypted into /dev/shm before the dry-run check, so a dry run also leaves a decrypted tokenizer directory behind.

Engine arguments follow -- and are appended after kLLM’s own arguments and the recipe’s arguments, so they override both:

$ kllm serve Qwen/Qwen3-8B --dry-run -- --context-length 32768
$ kllm serve ./Qwen3-8B-sealed --key --engine vllm -- --max-model-len 16384

kllm unlock

Help text:

decrypt a sealed model’s tokenizer into memory for another launcher (Dynamo, llm-d, …) and print its path

Synopsis:

kllm unlock MODEL [--key KEYFILE] [--engine {sglang,vllm}]

Unlocks the key, checks the model’s fingerprint, decrypts the permuted tokenizer into a new 0700 directory under /dev/shm and prints only that path on standard output; status lines go to standard error. This makes the command usable in a substitution.

For a model with locked weights, the command first unlocks the weights into the cache directory, as kllm serve --key does (Locked weights), and reports the unlocked directory on standard error in a line weights unlocked: PATH. The engine must then load that directory as the model, not the sealed directory:

$ TOK=$(kllm unlock ./Qwen3-8B-sealed --key model.key 2>unlock.log)
$ MODEL=$(sed -n 's/^ *weights unlocked: //p' unlock.log)
$ python -m sglang.launch_server --model-path "${MODEL:-./Qwen3-8B-sealed}" \
    --tokenizer-path "$TOK"

For a model sealed with --no-lock no such line is printed, and the engine loads the sealed directory.

MODEL
Help text:

sealed model directory

--key KEYFILE
Default:

~/.kllm/keys/default.key

Help text:

none

--engine {sglang,vllm}
Default:

sglang

Help text:

MiniMax H3 prints a model directory for sglang serve instead

For a MiniMax H3 pipeline the command prints the directory of an in-memory view of the pipeline with the decrypted tokenizer files in place; H3 requires sglang.

Warning

Nothing removes the decrypted directory when the engine exits. Remove it yourself, or use a memory-backed volume per container.

kllm verify

Help text:

check a sealed model against its public base: seal the base again under the key and compare tensor by tensor

Synopsis:

kllm verify MODEL --base BASE [--revision REVISION] [--subfolder SUBFOLDER] [--key KEYFILE] [--workdir WORKDIR] [--ulps ULPS]

Checks that a sealed model is exactly its public base sealed under the given key. The command unlocks the key, checks the model’s fingerprint, downloads the base when it is a Hugging Face repository, seals it again in a temporary directory under the same key and transform (and with the same client-keyed scheme and locking), and compares the two: every tensor (unlocked, when the weights are locked), the decrypted permuted tokenizer and every other file the seal writes. When the model directory carries a kllm-manifest.json, as catalog models do, it also checks every file against the manifest’s digests. The files that publication adds (README.md, kllm-manifest.json and the LICENSE, NOTICE and USE_POLICY files) are not compared with the second seal.

The command prints the number of tensors equal to the bit, the number within the allowed difference, and every tensor or file that differs, is missing or is unexpected. It exits with status 0 when everything matches and 1 otherwise. See Checking a published model.

MODEL
Help text:

sealed model directory

--base BASE
Default:

required

Help text:

the base: a directory or a Hub repo id

A local directory is used as it is. Any other value is downloaded from the Hugging Face Hub (safetensors only; HF_ENDPOINT and HF_TOKEN are honored), which requires the huggingface_hub package.

--revision REVISION
Default:

the base revision recorded in kllm-manifest.json, otherwise main

Help text:

of a Hub base (default: from kllm-manifest.json, else main)

--subfolder SUBFOLDER
Default:

none

Help text:

the model’s part of a Hub repo

For example FL2VA for a MiniMax H3 pipeline.

--key KEYFILE
Default:

~/.kllm/keys/default.key

Help text:

none

The key the model was sealed under. A key with a different fingerprint is refused.

--workdir WORKDIR
Default:

the system’s temporary directory

Help text:

for the download and the second seal

The directory must have room for the base and its second seal; both are removed when the command ends.

--ulps ULPS
Default:

1

Help text:

allowed difference of tensors rewritten in floating point

Tensors that sealing rewrites in floating point (under orth, and the latent interfaces of a client-keyed Z-Image pipeline) may differ by this many units in the last place, the difference another device or library version can introduce. Under perm every other tensor must be equal to the bit.

Note

The check needs the key, so only a key holder can run it. It shows that the files are what the open-source sealing code makes of the public base. It is not a signature and does not establish who published them.

kllm cloud

Help text:

copy the cloud’s part of a sealed model (no key, no tokenizer) for kllm serve --client

Synopsis:

kllm cloud MODEL -o OUTPUT [--key KEYFILE]

Writes the directory that the cloud serves in client-keyed mode, from a sealed model on the user’s machine, for example a catalog model. The output excludes trusted.kllm, every tokenizer file, every chat_template* file and the tokenizer/, processor/ and vae/ directories, as the -cloud directory of kllm seal --client does. When the weights are locked, the command unlocks them with the key on the way, because the keyless server cannot; the key is not copied. Otherwise files are hard-linked where possible, or copied. The command prints the number of files written and the cloud-side kllm serve command.

MODEL
Help text:

sealed model directory

-o OUTPUT, --output OUTPUT
Default:

required

Help text:

none

The directory must not exist.

--key KEYFILE
Default:

none

Help text:

unlock locked weights for the cloud (the key is not copied)

Required when the weights are locked; the command otherwise exits with <MODEL> is locked; unlock it for the cloud with --key. Not read for a model sealed with --no-lock.

Important

The cloud’s copy holds the sealed weights unlocked. A cloud that also holds the public base can match them to it and recover the transforms and the vocabulary permutation (Weights of a public model, or of a fine-tune of one). Client-keyed serving protects what the cloud records and relays, not the weights (Client-keyed mode).

kllm client

Help text:

the user’s side of a model served with kllm serve --client

Synopsis:

kllm client {proxy,generate} ...

Both subcommands run on the user’s machine, hold the key and the decrypted tokenizer in local memory, and talk to a cloud server started with kllm serve --client. They need the client extra; kllm client generate also needs the image extra.

kllm client proxy

Help text:

local OpenAI-compatible server that talks token ids to the cloud

Synopsis:

kllm client proxy MODEL --url URL [--key KEYFILE] [--engine {sglang,vllm}] [--host HOST] [--port PORT] [--name NAME]

Serves /health, /v1/models, /v1/chat/completions, /v1/completions, /generate (SGLang only) and /flush_cache locally. It applies the chat template and the permuted tokenizer on the user’s side and sends only token ids to the cloud.

MODEL
Help text:

the sealed directory (the user’s copy)

The sealed directory, not its -cloud export.

--url URL
Default:

required

Help text:

the cloud server, e.g. http://host:8000

--key KEYFILE
Default:

~/.kllm/keys/default.key

Help text:

none

--engine {sglang,vllm}
Default:

sglang

Help text:

none

Must match the engine of the cloud server.

--host HOST
Default:

127.0.0.1

Help text:

none

Warning

The proxy decodes any token ids it receives and has no authentication. Keep it on 127.0.0.1.

--port PORT
Default:

8001

Help text:

none

--name NAME
Default:

the name of the MODEL directory

Help text:

model name to report (default: dir name)

kllm client generate

Help text:

Z-Image: draw PROMPT, decoding the cloud’s latents locally

Synopsis:

kllm client generate MODEL PROMPT --url URL [--key KEYFILE] [-o OUTPUT] [--negative-prompt NEGATIVE_PROMPT] [--size SIZE] [--steps STEPS] [--guidance-scale GUIDANCE_SCALE] [--seed SEED] [--device DEVICE]

Encodes the prompt locally, requests latents from the cloud, decodes them with the public VAE on the local device and writes a PNG file. It prints the output path, the cloud time and the decode device. See Generating images in client-keyed mode.

MODEL
Help text:

the sealed directory (the user’s copy)

PROMPT
Help text:

none

Text prompt for the image.

--url URL
Default:

required

Help text:

the cloud server, e.g. http://host:8000

--key KEYFILE
Default:

~/.kllm/keys/default.key

Help text:

none

-o OUTPUT, --output OUTPUT
Default:

kllm.png

Help text:

none

--negative-prompt NEGATIVE_PROMPT
Default:

none

Help text:

none

--size SIZE
Default:

1024x1024

Help text:

none

Image size as WIDTHxHEIGHT.

--steps STEPS
Default:

28

Help text:

none

Number of denoising steps.

--guidance-scale GUIDANCE_SCALE
Default:

4.0

Help text:

none

--seed SEED
Default:

0

Help text:

none

--device DEVICE
Default:

cuda if available, otherwise mps, otherwise cpu

Help text:

for the VAE (default cuda, mps, cpu)

kllm demo

Help text:

two-stage keyed demo: sealed LLM -> sealed diffusion model

Synopsis:

kllm demo {up,down,run} ...

A sealed LLM expands an idea into an image prompt, and a sealed diffusion model draws it. Each stage runs twice, under the right key and under a random wrong key with the public base checkpoint, as an attacker holding the sealed files would run it.

Warning

In kLLM 0.1.0.dev0, kllm demo up fails when it starts the first LLM server. The demo is under repair; the options below are documented for reference only.

kllm demo up

Help text:

start the 4 demo servers (right and wrong key per stage)

Synopsis:

kllm demo up --llm LLM --llm-base LLM_BASE --image IMAGE --image-base IMAGE_BASE [--key KEYFILE] [--gpus GPUS] [--host HOST] [--port PORT] [--llm-mem-fraction LLM_MEM_FRACTION] [--timeout TIMEOUT]

Records its state in ~/.kllm/demo/state.json and writes one log per server to ~/.kllm/demo/. Before starting, it compares free GPU memory (from nvidia-smi) with a rough footprint of 29,000 MiB per LLM server and 25,000 MiB per image server.

--llm LLM
Default:

required

Help text:

sealed LLM directory

--llm-base LLM_BASE
Default:

required

Help text:

public base checkpoint of the LLM

--image IMAGE
Default:

required

Help text:

sealed diffusion pipeline directory

--image-base IMAGE_BASE
Default:

required

Help text:

public base pipeline

--key KEYFILE
Default:

~/.kllm/keys/default.key

Help text:

none

--gpus GPUS
Default:

the first two visible GPUs

Help text:

1-4 GPU indices: LLMs share the first and images the second when two are given; three give each LLM its own GPU (default: first two visible)

--host HOST
Default:

127.0.0.1

Help text:

none

--port PORT
Default:

30100

Help text:

base port; LLMs +1/+2, images +101/+102

--llm-mem-fraction LLM_MEM_FRACTION
Default:

0.35

Help text:

none

Passed to SGLang as --mem-fraction-static.

--timeout TIMEOUT
Default:

900

Help text:

none

Seconds to wait for all servers to become healthy.

kllm demo run

Help text:

turn an idea into images under right and wrong keys

Synopsis:

kllm demo run IDEA [-o OUTPUT] [--size SIZE] [--steps STEPS] [--seed SEED] [--max-tokens MAX_TOKENS]

Writes prompts.json, correct.png, wrong-image-key.png, wrong-text-key.png and the side-by-side compare.png to the output directory.

IDEA
Help text:

none

-o OUTPUT, --output OUTPUT
Default:

demo-out

Help text:

none

--size SIZE
Default:

1024x1024

Help text:

none

--steps STEPS
Default:

28

Help text:

none

--seed SEED
Default:

42

Help text:

none

--max-tokens MAX_TOKENS
Default:

200

Help text:

none

kllm demo down

Help text:

stop the demo servers

Synopsis:

kllm demo down

Sends SIGTERM to the process group of every server recorded in ~/.kllm/demo/state.json, then SIGKILL to those still running after 60 seconds, and removes the state file.

Warning

The command trusts the recorded process IDs. After a reboot, delete a stale ~/.kllm/demo/state.json instead of running kllm demo down.

Model, key and runtime commands (0.2 line)

Note

Part of the upcoming 0.2 line. The commands in this section are present on the main branch and are not part of a tagged release; the source tree still carries the development version 0.1.0.dev0. Their options and output may change before 0.2.0. The commands described above are unchanged by them.

The commands below add a local model library, a key ring and background servers to the existing keygen, seal and serve commands. They are peripheral: every one of them reuses the existing sealing and serving code, and none of them changes how a model is sealed or how kllm serve runs. Task-oriented descriptions are in Quickstart, Model library and naming and Model keys.

Common elements of these commands:

  • Model references. Every REF argument accepts a library alias such as qwen3 or qwen3:8b, a Hugging Face repository (hf://org/repo, hf://org/repo@revision or org/repo), a ModelScope repository (ms://org/repo) or a local directory. The forms are described in Model references.

  • kLLM home. Models, keys, server records and logs live under KLLM_HOME (default ~/.kllm) in the subdirectories models/, keys/, run/ and logs/, created with mode 0700. See Settings.

  • Key names. Options named --key NAME take the name of a key in the key ring (KLLM_HOME/keys/NAME.key) or the path of a key file.

  • Output. Output is plain text. Results are written to standard output and progress to standard error, so that, for example, kllm list --quiet composes with other tools. A failure prints an error: line and, where one applies, a hint: line on standard error, and exits with status 1. An interrupted command (Ctrl-C) exits with status 130. Command-line errors exit with status 2, as described in General behavior.

kllm pull

Synopsis:

kllm pull REF [--revision REVISION] [--force]

Downloads a model into the local store. Hugging Face repositories are downloaded with huggingface_hub (the command prints an installation hint when the package is missing); ModelScope repositories are downloaded when the modelscope package is installed. Only the files needed for serving are fetched. The download goes to a temporary directory next to its destination, after a check of the free disk space, and is renamed into place only when it is complete; an interrupted download resumes on the next attempt. The command then writes a store record next to the model directory (Store record and manifests), with the SHA-256 digest of every file, the source and the revision. When the repository publishes a catalog manifest, kllm-manifest.json, as official sealed models do, exactly the listed files are downloaded and checked against it, a mismatch stops the command, and the manifest is kept unchanged. A pull never loads weights. For a sealed model, the command reports whether the weights are locked and which key in the key ring opens the model, or how to obtain its model key. A catalog model that is still rolling out and not yet published is reported as such; a held one is refused.

REF

The model to download. A library alias resolves to a pinned repository and revision.

--revision REVISION
Default:

the revision pinned by the library, or the default branch of the repository

A branch, tag or commit to download.

--force
Default:

off

Download the model again and replace the stored copy. A model that is being served is refused; stop its server first.

kllm run

Synopsis:

kllm run REF [PROMPT ...] [--engine {sglang,vllm}] [--key NAME] [--port PORT] [--tp TP] [--timeout SECONDS] [--rm]

Runs a language model locally and opens a chat session, or answers one prompt and exits. The command pulls the model if it is not in the store, reuses a running server for the same model or starts one in the background, waits until the server answers on /v1/models, and then streams the replies. If the server process exits while it loads, the command stops waiting and prints the end of its log. Image and video models are refused with the matching kllm serve command, because kllm run chats with language models only.

The background server is kllm serve started as a separate process in its own session, bound to 127.0.0.1. Its process ID, process start time, launch ID and port are recorded in KLLM_HOME/run/servers.json and its output goes to KLLM_HOME/logs/. It keeps running after the session ends, until kllm stop stops it, unless --rm is given. A server that fails to start, or whose start is interrupted, is stopped.

For a sealed model, the key is the one named by --key; without the option, the key whose fingerprint matches the model is taken from the key ring. When no key matches, the command exits and names kllm key import. The passphrase is read on the terminal, or from KLLM_PASSPHRASE, by kllm run itself and checked against the key file before the server starts. For a catalog model without a matching key, the message names the application page for model keys and the kllm key import command. It is then handed to the server process only through that process’s environment. It never appears on a command line, in a file or in a log. Processes of the same user can read a process’s environment; run kllm run only on a machine whose other processes under the same account are trusted. For a plain model, KLLM_PASSPHRASE is removed from the server’s environment, and --key is refused.

Important

The background server unlocks a sealed model as kllm serve --key does: it decrypts the permuted tokenizer into /dev/shm on the local machine and unlocks locked weights into the cache directory (Locked weights), and kLLM removes neither copy. The first start of a locked model takes longer and needs free disk space for the unlocked weights. kllm run therefore protects nothing from the machine it runs on. It is intended for the machine of the key’s owner. For serving on a machine operated by someone else, see Deployment profiles.

In a chat session:

Input

Effect

/bye or Ctrl-D

End the session.

/clear

Clear the conversation history of the session.

/help

List the session commands.

"""

Begin or end a multi-line message.

Ctrl-C

Cancel the reply being generated.

The conversation history is kept in memory only.

REF

The model to run.

PROMPT
Default:

none

A prompt to answer once. The reply is written to standard output and the command exits. Text piped on standard input is answered the same way. Without either, the command opens a chat session.

--engine {sglang,vllm}
Default:

the first installed engine among those the library lists for the model

The engine of a newly started server.

--key NAME
Default:

the key in the key ring whose fingerprint matches the model

The key of a sealed model: a key name or a key file path. A key whose fingerprint differs from the model’s is refused.

--port PORT
Default:

the first free port from 11400 upward

The port on 127.0.0.1 of a newly started server.

--tp TP
Default:

the engine’s default

The tensor-parallel size of a newly started server.

--timeout SECONDS
Default:

1800

How long to wait for the engine to load the model.

--rm
Default:

off

Stop the server when the session ends or the one-shot answer is complete. A server that was already running before the command started is left running.

kllm list

Synopsis:

kllm list [-q] [--json]

Lists the models in the local store with their name, size, whether each is sealed (with its transform, key fingerprint and, for locked weights, locked), and the time it was last modified. kllm ls is an alias.

-q, --quiet

Print the names only, one per line.

--json

Print the list as JSON.

kllm ps

Synopsis:

kllm ps [--json]

Lists the background servers started by kllm run, with their model, engine, port, process ID, status (starting or ready), uptime and address. Records whose process has exited, or whose process ID now belongs to another process, are dropped.

--json

Print the list as JSON.

kllm stop

Synopsis:

kllm stop REF|all

Stops the background server of a model, or every recorded server with all. The command signals only the process group recorded for the server, and only after it has checked that the recorded process start time still matches, so that a recycled process ID is never signalled. It sends SIGTERM and, if the group has not exited after a timeout, SIGKILL. No process is ever selected by name or pattern. Process checks read /proc and are available on Linux only.

kllm rm

Synopsis:

kllm rm REF [REF ...]

Removes models from the local store. A model that is being served is refused; stop its server first. Directories outside the store are never removed, and key files are never removed by this command. Unlocked copies of a locked model in the cache directory (Locked weights) are not removed either; the command prints where they are.

kllm show

Synopsis:

kllm show REF [--json]

Prints what is known about a model, from the local store, the library or both. For a stored model: its name, local path, size, source and revision, the model it was sealed from, whether it is sealed, and for a sealed model its transform, scheme, whether its weights are locked, its key fingerprint and the name of the matching key in the key ring, if any, or how to obtain the model key. For a catalog model it adds the base and its revision, the base license, the key version and the kLLM version that sealed it. For a library entry: its source and pinned revision, architecture, modality, license, download size, minimum GPU memory, engines and seal transforms, and for an official sealed model its status, the base it was sealed from and the license notice. The information comes from the store record, the catalog manifest, the sealed model’s config.json and the library; no key or passphrase is needed.

--json

Print the information as JSON.

kllm check

Help text:

re-hash a stored model against its store record and, for an official sealed model, its catalog manifest; or check an https:// gateway’s attestation binding

Synopsis:

kllm check MODEL|URL [--cafile CAFILE]

For a model reference, recomputes the SHA-256 digest of every file and compares the result with the store record and, for an official sealed model, with its catalog manifest (kllm-manifest.json); it also reports a catalog model whose seal names another key than its manifest. Missing, altered and unexpected files are reported, and the command exits with status 1 on any difference. For a catalog model, the command then prints the kllm verify command that checks the weights against the public base.

For an https:// address, the command runs the attestation binding check python -m kllm.keyed.attest against that address (Module entry points) and exits with its status. KLLM_PASSPHRASE is removed from the environment of that check, which needs no key.

Earlier builds of the 0.2 line named this command kllm verify. That name now belongs to the core command that checks a sealed model against its public base (kllm verify).

--cafile CAFILE
Default:

none

Help text:

CA bundle for an https:// gateway

A CA bundle for the gateway certificate, passed on to the binding check. It applies to an https:// address only.

Important

A manifest check shows that the local files are those recorded at download or sealing time, and, for official models, those published with the artifact. It needs no key and says nothing about how the files relate to the public base; kllm verify checks that. It is not a signature and says nothing about who produced the artifact. The binding check for an https:// address does not verify the quote’s signature chain, TCB status or measurements; see Verifying a deployment.

kllm create

Synopsis:

kllm create NAME --from REF --key NAME [--transform {orth,perm}] [--no-lock]

Seals a plain model from the store under a private key and stores the result as a new local model. The command runs the existing kllm seal SOURCE -o OUTPUT --key KEYFILE --transform TRANSFORM as a separate process, which asks for the key passphrase itself (or reads KLLM_PASSPHRASE), so sealing behaves exactly as described in kllm seal. The result is recorded with a manifest that names the model it was sealed from. Its weights are locked, as kllm seal locks them by default. The result is sealed for key-holding serving (kllm serve --key or kllm run); for client-keyed serving, use kllm seal with --client directly.

NAME

The local name of the sealed model, in name[:tag] form. A name without a tag gets the tag latest. Names of library models are refused, as is a library family name without a tag.

--from REF
Default:

required

The plain model to seal. It must already be in the store; pull it first. A model that is already sealed is refused.

--key NAME
Default:

required

The key to seal under: a key name or a key file path.

--transform {orth,perm}
Default:

orth where the library lists it for the model; otherwise the first listed transform (perm for FP8 checkpoints)

The transform, as for kllm seal. A transform the library does not list for the model is refused. See Choosing a transform.

--no-lock
Default:

off

Passed to kllm seal: leave the sealed weights readable instead of locking them (kllm seal).

Sealing needs a validated implementation for the model’s architecture. A model whose architecture is not supported is refused by kllm seal; request support as described in Requesting model support.

kllm key

Synopsis:

kllm key {new,list,import,rm} ...

Manages the key ring in KLLM_HOME/keys. Each key is a passphrase-encrypted key file in the format written by kllm keygen, stored as NAME.key with mode 0600. Names consist of lowercase letters, digits, ., _ and -.

kllm key new NAME [--force]

Creates a new key under NAME with the existing key-file code, reads its passphrase twice on the terminal (or once from KLLM_PASSPHRASE) and prints its fingerprint. An existing key is replaced only with --force. With KLLM_HOME unset, the key named default is the file ~/.kllm/keys/default.key, which is also the default key of the existing --key options.

kllm key list [-q] [--json]

Lists the keys with their names, fingerprints, permissions and paths. Fingerprints are read from the public header of each key file; no passphrase is needed. -q prints the names only.

kllm key import NAME FILE [--force]

Copies an existing key file into the key ring under NAME with mode 0600, after checking its format. Symbolic links are refused, and an existing key is replaced only with --force. A key received from the kLLM team, such as the model key of a catalog model, is added this way: kllm key import qwen3-8b-kllm Qwen3-8B-kllm.key.

kllm key rm NAME [--yes]

Removes a key from the key ring after the key name is typed again to confirm, or without confirmation with --yes.

Warning

Removing or overwriting a key file destroys the key unless a copy exists elsewhere. Models sealed under it can then no longer be served.

kllm doctor

Synopsis:

kllm doctor [--json]

Checks the host before a model is pulled or served and prints one line per check with its status (ok, warn or fail) and, where one applies, a suggested fix: the Python version, the NVIDIA driver and the visible GPUs, the CUDA version, free disk space in KLLM_HOME, the size of /dev/shm, the recorded background servers and the first server port, the permissions of the kLLM home, the key directory and key files, and the installed engine and download packages. Every check is read-only. The command exits with status 1 when any check fails.

--json

Print the checks as JSON.

kllm request

Synopsis:

kllm request MODEL_ID

Prints the address of a new Model request issue in the GitHub repository with the model identifier filled in, for example https://github.com/ANetResearch/kLLM/issues/new?template=model-request.yml&title=...&model=org/model&upstream=.... For a Hugging Face identifier of the form org/model, the upstream link is filled in as well. The command does not open a browser and sends nothing. See Requesting model support.

kllm logs

Synopsis:

kllm logs REF [-n LINES] [-f]

Prints the end of the log of the background server of a model from KLLM_HOME/logs/. When the model has no running server, the most recent log of the model is printed.

-n LINES, --lines LINES
Default:

100

The number of lines to print.

-f, --follow

Keep printing new lines until the server exits or the command is interrupted.

kllm completion

Synopsis:

kllm completion {bash,zsh,fish}

Prints a shell completion script generated from the command-line parser. Model and key names are completed from kllm list --quiet and kllm key list --quiet. For example, for bash:

$ kllm completion bash > ~/.local/share/bash-completion/completions/kllm

Planned commands

Note

Planned — not available. A resident local service on 127.0.0.1:11400 that routes OpenAI-compatible requests by model, an Ollama-compatible /api/chat and /api/generate interface, keep-alive and admission control, a local web console and Modelfile-style recipes. See Roadmap.

Module entry points

The following modules are run with python -m. They are not kllm subcommands.

python -m kllm.keyed.attest

Description:

check a kLLM gateway’s attestation binding

Synopsis:

python -m kllm.keyed.attest [--cafile CAFILE] url

Fetches /attestation from an attesting gateway with a fresh nonce and checks that the evidence names the TLS key of the connection. It exits with status 0 if the binding holds and 1 otherwise. See Verifying a deployment.

url
Help text:

https://host:port of the gateway

--cafile CAFILE
Default:

none

Help text:

none

CA bundle for the gateway certificate. Without it, certificate and host name checks are disabled, because a self-signed certificate is vouched for by the quote.

Important

This is a binding check only. It does not verify the quote’s signature chain, TCB status or measurements; do that with the Trustee attestation service, Intel DCAP QVL or AMD snpguest. A response whose source is none (no TEE) passes the binding check; the command then prints a notice that the server runs outside a confidential VM.

python -m kllm.kernels.h3

Prints the directory to put on PYTHONPATH so that every SGLang process loads the INT8 kernels selected by KLLM_INT8. kllm serve --int8 sets both for a sealed H3 pipeline. It takes no options.

python -m kllm.keyed.client.server

The keyless Z-Image latent server that kllm serve --client starts for a client-keyed Z-Image pipeline. It accepts --model (required), --host (default 127.0.0.1), --port (default 30010), --device (default cuda), --size (default 1024x1024), --steps (default 28) and --guidance-scale (default 4.0). When it is started through kllm serve, the host and port of kllm serve apply. Start it through kllm serve rather than directly.