Serving with vLLM

kllm serve starts SGLang by default. --engine vllm starts vLLM instead, for plain models, sealed models and client-keyed models alike. This page describes what changes with vLLM, what is not available, and what was measured.

Installation

Install kLLM with the vllm extra (vllm>=0.11) into the environment that will run the engine. Add seal for keys and sealed models, and client on the user’s machine for client-keyed serving:

$ git clone https://github.com/ANetResearch/kLLM
$ cd kLLM
$ pip install -e '.[seal,vllm]'

kllm serve --engine vllm runs the vllm console script installed next to the Python interpreter that runs kllm. The published measurements used vLLM 0.31.0. See Installation guide for the other extras and for the container image built with --build-arg BASE=vllm/vllm-openai:latest.

Serving a model

$ kllm serve Qwen/Qwen3-8B --engine vllm                       # plain model
$ kllm serve Qwen3-8B-sealed --key --engine vllm               # sealed model
$ kllm serve Qwen3-8B-sealed --key quickstart.key --engine vllm --tp 2 -- --max-model-len 8192

--dry-run prints the command that kllm serve would run, without starting it:

$ kllm serve Qwen/Qwen3-8B --engine vllm --tp 2 --dry-run -- --max-model-len 8192
kLLM 0.1.0.dev0 · backend vllm · 2×A40
  recipe  vllm defaults
  serving http://0.0.0.0:8000/v1 (OpenAI compatible)
/path/to/venv/bin/vllm serve Qwen/Qwen3-8B --tensor-parallel-size 2 --host 0.0.0.0 --port 8000 --max-model-len 8192

The engine command differs from SGLang’s as follows:

`kllm serve` input	SGLang	vLLM
Launcher	`python -m sglang.launch_server --model-path MODEL`	`vllm serve MODEL`
`--tp N`	`--tp N`	`--tensor-parallel-size N`
Sealed model’s tokenizer (`--key`)	`--tokenizer-path DIR`	`--tokenizer DIR`
`--host`, `--port`	passed unchanged	passed unchanged
Built-in recipes	consulted unless `--no-recipe`	never consulted; `--recipe` and `--no-recipe` have no effect
Arguments after `--`	appended last	appended last

Without --tp, the tensor-parallel size is the number of visible GPUs (at least 1). The three built-in recipes are measured on SGLang only, so a model that would match a recipe under SGLang runs with vLLM’s defaults.

With --key, kllm serve unlocks the key, decrypts the permuted tokenizer into a 0700 directory under /dev/shm and passes that directory to vLLM. The model id reported by /v1/models is the path given to kllm serve.

Warning

Outside a confidential VM, kllm serve --key decrypts the tokenizer into host /dev/shm and offers no protection from the host; it is a development mode. The directory is not removed when the engine exits. See Deployment profiles.

Client-keyed serving

Client-keyed mode works with vLLM on both sides. In the cloud, kllm serve --client starts vLLM with --skip-tokenizer-init, so the engine accepts and returns token ids and never loads a tokenizer. On the user’s machine, kllm client proxy must be told the engine, because the two engines expose different token-id APIs:

$ kllm serve Qwen3-8B-sealed-cloud --client --engine vllm --host 127.0.0.1    # cloud
$ kllm client proxy Qwen3-8B-sealed --url http://127.0.0.1:8000 --engine vllm  # user

With --engine vllm the proxy sends permuted prompt ids to vLLM’s /v1/completions with return_token_ids and decodes the returned ids locally. It serves /v1/chat/completions and /v1/completions on 127.0.0.1:8001. SGLang’s native /generate endpoint is not available through the proxy with vLLM. The logprobs, prompt_logprobs and echo fields of /v1/completions are passed through only with vLLM; their token ids are the permuted ids and are not mapped back.

vLLM’s request log records sampling parameters only, under either --key or --client. See Client-keyed mode for the full procedure and its security scope.

Other launchers

For an orchestrator that starts vLLM itself, kllm unlock decrypts the tokenizer and prints its directory on standard output:

$ TOK=$(kllm unlock Qwen3-8B-sealed --key quickstart.key --engine vllm)
$ vllm serve Qwen3-8B-sealed --tokenizer "$TOK"

The process that holds the decrypted tokenizer belongs in the trusted domain (Trusted domain). See NVIDIA Dynamo and Kubernetes and llm-d.

Not available with vLLM

Feature	Behavior with `--engine vllm`
MiniMax H3 pipelines	Refused with `MiniMax H3 pipelines are served with SGLang`.
INT8 DiT kernels (`--int8`)	Apply to sealed MiniMax H3 only, which is served with SGLang.
Built-in recipes	Not consulted.
Client-keyed Z-Image	Served by kLLM’s own latent server, not by an engine; `--engine` does not apply.

Measured results

Qwen3-1.7B under vLLM 0.31.0 on one NVIDIA A40 48 GB, bf16, random token ids of 1024 input and 128 output tokens, concurrency 1 with 10 requests and concurrency 16 with 128 requests, GPU shared with another tenant’s light job, single runs (ENGINE=vllm benchmarks/llm.sh):

	plain	`perm`	`orth`
Median TTFT, concurrency 1 (ms)	65.3	66.3	68.4
Median TPOT, concurrency 1 (ms)	6.59	6.60	6.59
Output throughput, concurrency 16 (tok/s)	1097	1094	1095
Median TPOT, concurrency 16 (ms)	12.32	11.97	11.94
Perplexity	5.4990	5.4970	5.5066
Mean \|Δ log-prob\| against plain		0.061	0.090

Other vLLM rows, under the same conditions (output throughput at concurrency 16, perplexity, and mean |Δ log-prob| against plain):

Model	plain	`perm`
Qwen3-1.7B-FP8 (128x128 blocks)	605 tok/s · 5.7770	628 tok/s · 5.7815 · 0.058
Qwen3-30B-A3B-FP8 (MoE, 128 experts), TP=2 on two A40	359 tok/s · 1.3424	378 tok/s · 1.3428 · 0.033

The higher sealed figures in these rows are attributed to noise on the shared host. The A40 has no FP8 tensor cores, so vLLM runs FP8 checkpoints through weight-only Marlin kernels.

Client-keyed serving of Qwen3-1.7B (perm) through kllm client proxy cost 7% output throughput at concurrency 16 against --key serving under vLLM (1036 to 963 tok/s); under SGLang it cost none. See Client-keyed mode for the full table.

Standard evaluations of Qwen3-1.7B under vLLM with lm-evaluation-harness, full test sets (benchmarks/eval.sh):

	plain	`perm`	`orth`
MMLU (accuracy, 0-shot)	0.5560	0.5562	0.5548
GSM8K (exact match, 5-shot, strict)	0.6945	0.6952	0.6960
GSM8K (exact match, 5-shot, flexible)	0.6975	0.6960	0.6975

Standard errors are 0.004 for MMLU and 0.013 for GSM8K; all differences are within one standard error.

Note

tools/ppl.py scores every request on a cold prefix cache. Under SGLang it flushes the cache before each request; under vLLM, start the engine with --no-enable-prefix-caching instead. benchmarks/llm.sh passes this flag. See Benchmarking plain and sealed serving.