NVIDIA Dynamo

NVIDIA Dynamo starts and routes the engine workers itself, so kllm serve is not in the launch path. A sealed model needs two things from kLLM under Dynamo: the path of its decrypted tokenizer, which kllm unlock writes to memory and prints, and, for a model with locked weights, the unlocked copy of the weights, which the same command writes to its cache directory. The workers then run the sealed checkpoint as a stock model, and Dynamo’s features (disaggregated prefill and decode, KV-aware routing, the planner) operate on it unchanged, because the GPU sees an ordinary checkpoint of the original architecture. This page describes kllm unlock, the launch of the workers, the placement of components that hold the tokenizer, and cleanup.

Warning

This setup has not been run end to end against a Dynamo release. The commands below are taken from examples/dynamo/README.md. The Dynamo module names and options shown are those of the example; confirm them against the documentation of the Dynamo release in use.

Note

Planned — on the Roadmap, not available in 0.1.0.dev0. Dynamo and llm-d setups that run end to end and are kept in continuous integration.

The kllm unlock command

kllm unlock MODEL [--key KEYFILE] [--engine {sglang,vllm}]

Argument

Default

Meaning

MODEL

required

The sealed model directory.

--key KEYFILE

~/.kllm/keys/default.key

The key file the model was sealed with. Its passphrase is read from the terminal, or from KLLM_PASSPHRASE.

--engine

sglang

The engine the path is meant for. For language models the output is the same for both engines. For MiniMax H3, which is served with SGLang only, the command prints a model directory for sglang serve instead of a tokenizer directory, and --engine vllm is refused.

The command unlocks the key file, checks that the model was sealed with a key of the same fingerprint, unlocks locked weights into the cache directory (Locked weights), and decrypts the permuted tokenizer into a fresh directory with mode 0700 under /dev/shm (or the default temporary directory when /dev/shm does not exist). It prints only the tokenizer path to standard output; status lines go to standard error, among them weights unlocked: PATH with the unlocked directory of a locked model. This makes it suitable for command substitution:

$ TOK=$(kllm unlock /models/Qwen3-8B-sealed --key model.key 2>unlock.log)
$ MODEL=$(sed -n 's/^ *weights unlocked: //p' unlock.log)
$ MODEL=${MODEL:-/models/Qwen3-8B-sealed}

For a model sealed with --no-lock no such line is printed, and the workers load the sealed directory itself.

The command refuses a model that is not sealed, a key with a different fingerprint and a client-keyed Z-Image pipeline. It is also refused when KLLM_PRODUCTION is set to a value other than empty or 0, because it takes the key from a key file; see Confidential VM deployment.

Launching the workers

Unlock the model, start the Dynamo frontend, and start an SGLang or a vLLM worker on the unlocked weights with the unlocked tokenizer:

export KLLM_PASSPHRASE=...        # development only; see the warning below
TOK=$(kllm unlock /models/Qwen3-8B-sealed --key model.key 2>unlock.log)
MODEL=$(sed -n 's/^ *weights unlocked: //p' unlock.log)
MODEL=${MODEL:-/models/Qwen3-8B-sealed}

# frontend (OpenAI-compatible API and router)
python -m dynamo.frontend --http-port 8000 &

# SGLang worker
python -m dynamo.sglang --model-path "$MODEL" --tokenizer-path "$TOK" \
  --served-model-name Qwen3-8B --tp 1 &

# or a vLLM worker
python -m dynamo.vllm --model "$MODEL" --tokenizer "$TOK" \
  --served-model-name Qwen3-8B &

The worker receives the sealed weights, unlocked, through its model option and the secret tokenizer through its tokenizer option. No kLLM code runs inside the worker, and the key is not passed to it. A worker given the sealed directory of a locked model loads the locked weights, which compute nothing useful.

Placing components in the trusted domain

The unlocked tokenizer is the main secret at run time: it maps text to the permuted token ids that the sealed model expects. The unlocked weights are in the clear as well, in the cache directory and in GPU memory; for a public model, or a fine-tune of one, they reveal the transforms and the vocabulary permutation to a holder of the public base (Weights of a public model, or of a fine-tune of one). Every process that holds it can read requests and responses in clear text. Under Dynamo this includes more than the worker:

  • The worker’s tokenizer process, which loads the tokenizer from $TOK.

  • The frontend. Dynamo’s frontend tokenizes requests itself, for KV-aware routing, with the tokenizer that the worker registers. It then holds the secret vocabulary in the same way as the worker’s tokenizer process.

Keep the frontend in the trusted domain (Trusted domain): in the same confidential VM as the workers, or in a VM that is attested in the same way. A frontend on an untrusted host defeats sealing.

Warning

kllm unlock with a key file and KLLM_PASSPHRASE on an ordinary host is a development setup. The key file, its passphrase, the decrypted tokenizer in host memory and the unlocked weights on the host’s file system are then all readable by the host. kLLM 0.1.0.dev0 has no open-source path that delivers the key from a key broker to kllm unlock. See Deployment profiles.

Cleaning up

kLLM does not remove the decrypted tokenizer directory or the unlocked weights. Delete them after the workers stop, when the model is no longer served:

$ rm -rf "$TOK"
$ [ "$MODEL" != /models/Qwen3-8B-sealed ] && rm -rf "$MODEL"

See also