Benchmarking plain and sealed serving

The benchmarks/ directory of the repository holds one script per published table. Each script seals the model under a fresh key unless a sealed copy exists, serves the plain and sealed variants, and writes everything it measured to an output directory. This page describes how to run the scripts on other hardware, which script produced which published table, and the full LLM tables.

Note

kLLM has no benchmark command. Benchmarks are the shell and Python scripts described here; they are not part of the installed package or the container image, so run them from a source checkout.

Measurement conditions

Unless a table states otherwise, the published results were measured as follows:

one NVIDIA A40 48 GB per server (two for tensor parallelism 2);
SGLang 0.5.21 and vLLM 0.31.0, bf16 unless the checkpoint is FP8;
random token ids, 1024 input and 128 output tokens; concurrency 1 with 10 requests and concurrency 16 with 128 requests;
variants served one after another on the same GPU, with --no-recipe;
a GPU shared with another tenant’s light job;
single runs; no variance is reported.

Report new numbers with the GPU model, the engine version and these settings. Differences of a few percent between single runs on a shared host are not significant.

Prerequisites

A source checkout with kLLM installed with seal and the engine extra, serve or vllm.
Local checkpoint directories; the scripts take paths, not Hub ids.
For perplexity: the GPL-3, GFDL-1.3 and Apache-2.0 license texts under /usr/share/common-licenses, as shipped by Debian and Ubuntu.
For eval.sh: pip install 'lm_eval[api]' transformers.
For the H3 scripts: an SGLang build with diffusion support and PyAV; for h3_int8.sh also the int8 extra.
For zimage_client.py: the image extra.

The scripts create a key file in the output directory when none is given, with the passphrase from KLLM_PASSPHRASE (default kllm-bench). Use such keys for benchmarking only.

LLM latency, throughput and perplexity

$ benchmarks/llm.sh ~/models/Qwen3-8B                       # SGLang, GPU 0
$ ENGINE=vllm GPU=2 benchmarks/llm.sh ~/models/Qwen3-8B     # vLLM, GPU 2
$ python benchmarks/report.py out/Qwen3-8B-sglang

llm.sh serves plain, then each transform in TRANSFORMS. Each variant is scored by tools/ppl.py on the three license texts (the plain run is saved, and the sealed runs are compared with it token by token), then load-tested with sglang.bench_serving on random token ids when SGLang is importable, or with vllm bench serve otherwise. Engine flags are --mem-fraction-static 0.55 for SGLang, and --gpu-memory-utilization 0.55 --max-model-len 8192 --no-enable-prefix-caching for vLLM. report.py prints a Markdown table of median TTFT and TPOT at concurrency 1, output throughput and median TPOT at concurrency 16, perplexity and mean |Δ log-prob| against plain; it orders the sealed columns alphabetically.

`llm.sh` settings
Variable	Default	Meaning
`GPU`	`0`	Value of `CUDA_VISIBLE_DEVICES`.
`PORT`	`31040`	Engine port.
`ENGINE`	`sglang`	`sglang` or `vllm`.
`TRANSFORMS`	`perm orth`	Sealed variants. Use `perm` for FP8 checkpoints.
`KEY`	`OUT/bench.key`	Key file; created if missing.
`OUT`	`out/<model>-<engine>`	Output directory.
`EXTRA`	empty	Further engine flags, for example tensor-parallel options.

For tensor parallelism, select several GPUs with GPU, for example GPU=0,1 ENGINE=vllm. kllm serve sets the tensor-parallel size to the number of visible GPUs when --tp is not given.

Standard evaluations

$ benchmarks/eval.sh ~/models/Qwen3-1.7B                    # GSM8K and MMLU, vLLM, GPU 0
$ TASKS=gsm8k LIMIT=200 ENGINE=sglang benchmarks/eval.sh ~/models/Qwen3-1.7B

eval.sh runs lm-evaluation-harness against the OpenAI-compatible endpoint of each variant. The harness sends text and, for a sealed model, counts tokens with the tokenizer that kllm unlock decrypts, which is the one the server uses: an evaluator that holds the key is inside the trusted domain. Settings: GPU (0), PORT (31050), ENGINE (vllm), TRANSFORMS (perm orth), TASKS (gsm8k,mmlu), LIMIT, KEY, OUT (out/eval-<model>-<engine>) and EXTRA.

Client-keyed serving

$ benchmarks/client.sh ~/models/Qwen3-1.7B                  # SGLang, GPU 0
$ ENGINE=vllm GPU=3 benchmarks/client.sh ~/models/Qwen3-1.7B

client.sh seals with --client (perm by default), then serves the sealed model once with --key and once without a key from its -cloud directory behind kllm client proxy, one after the other on one GPU. Both runs are scored by tools/ppl.py (the client run against the keyed one), load-tested through the proxy, and asked one chat prompt with request logging at its most verbose. The script then searches both engine logs for three-word phrases of the prompt and the reply. Settings: GPU, PORT (31060; the proxy uses PORT+2), ENGINE, TRANSFORM, KEY, OUT and EXTRA. The results are in Client-keyed mode.

For Z-Image, start the latent server separately and run the comparison script:

$ kllm seal ~/models/Z-Image -o ~/models/Z-Image-client --client
$ CUDA_VISIBLE_DEVICES=0 kllm serve ~/models/Z-Image-client-cloud --client --host 127.0.0.1 --port 30010
$ CUDA_VISIBLE_DEVICES=1 python benchmarks/zimage_client.py ~/models/Z-Image ~/models/Z-Image-client \
    --key ~/.kllm/keys/default.key --url http://127.0.0.1:30010 --seeds 0 1

For three fixed prompts per seed, from the same noise, it runs the plain pipeline, the plain pipeline with the DiT’s attention on PyTorch’s math kernel (the noise floor), and the client through the keyless server, and prints latent error and PSNR against plain and the end-to-end times. Options: -o/--out (out/zimage-client), --size (1024), --steps (28), --guidance-scale (4.0), --seeds (0). The results are in Generating images in client-keyed mode.

MiniMax H3

$ benchmarks/h3.sh ~/models/MiniMax-H3/FL2VA                # 49 steps, GPUs 0 and 1
$ STEPS=8 GPUS=2,3 benchmarks/h3.sh ~/models/MiniMax-H3/FL2VA

h3.sh runs plain and sealed FL2VA at the same time on two GPUs of one host, so that both see the same contention. The sealed server draws the plain server’s noise, permuted into the sealed latent basis, and both save their final latents. The script prints the end-to-end times, the latent drift and the frame PSNR and audio SNR, and writes a side-by-side still to OUT/grid.png. For the bf16 noise floor, run the plain model against itself with --attention-backend torch_sdpa on one side. Settings: GPUS (0,1), PORTS (30021,30022), TRANSFORM (perm), STEPS (49), SEED (7), KEY, SEALED and OUT.

h3_int8.sh measures the INT8 DiT kernels on plain H3; see Running the H3 diffusion transformer on INT8 kernels. Settings: GPU (0), PORT (30031), STEPS (49), SEED (7), VARIANTS (plain int8 sdpa), INT8 (all, attention or linear) and OUT. The results of both scripts are in Sealing and serving MiniMax H3 and Running the H3 diffusion transformer on INT8 kernels.

Attacks

$ python tools/attacks/weights.py --base ~/models/Qwen3-8B --transform perm
$ python tools/attacks/tokens.py --help

See Attacks for the procedure and the results.

Which script produced which result

Published result	Command	Notes
LLM latency, throughput and perplexity, SGLang and vLLM	`benchmarks/llm.sh`, `benchmarks/report.py`	One model and engine per run; tables combined by hand.
fp16 log-probability comparison (0.053, 0.008, 0.032)	`benchmarks/llm.sh`	Not scripted as such; serve in fp16 through the engine’s dtype option in `EXTRA`.
MMLU and GSM8K	`benchmarks/eval.sh`
Client-keyed LLM table and log check	`benchmarks/client.sh`	One engine per run.
Client-keyed Z-Image fidelity	`benchmarks/zimage_client.py`	Server started separately.
H3 end-to-end time, latent drift, PSNR	`benchmarks/h3.sh`	One step count per run; the SDPA row needs a manual run.
INT8 time per step and fidelity	`benchmarks/h3_int8.sh`	Reports total denoising time.
Weight recovery and token attacks	`tools/attacks/weights.py`, `tools/attacks/tokens.py`
INT8 per-layer timings, Z-Image VAE decode times, sealing times	none	Measured outside the scripts.

Results: SGLang

One NVIDIA A40 48 GB, SGLang 0.5.21, bf16, under the conditions above (benchmarks/llm.sh):

	Qwen3-1.7B plain	`perm`	`orth`	Qwen3-8B plain	`perm`	`orth`
Median TTFT, concurrency 1 (ms)	98.0	105.2	104.3	199.5	200.1	200.4
Median TPOT, concurrency 1 (ms)	12.44	12.54	12.32	25.12	25.12	25.16
Output throughput, concurrency 16 (tok/s)	563	577	580	333	333	332
Median TPOT, concurrency 16 (ms)	21.5	22.7	22.4	35.3	36.2	37.3
Perplexity	5.4991	5.5057	5.5315	1.9050	1.9036	1.9063
Mean \|Δ log-prob\| against plain		0.069	0.098		0.031	0.045

Perplexity is over 14690 tokens of the GPL-3, GFDL-1.3 and Apache-2.0 license texts, which Qwen3-8B has largely memorized. For scale, serving plain Qwen3-1.7B in fp16 instead of bf16 moves log-probabilities by 0.053 on average; served in fp16, perm moves them by 0.008 and orth by 0.032. The vLLM table for Qwen3-1.7B is in Serving with vLLM.

Results: other families

Same texts and load. Each cell gives output throughput at concurrency 16, perplexity, and for sealed variants the mean |Δ log-prob| against plain.

Model	Engine	plain	`perm`	`orth`
Qwen2.5-1.5B-Instruct (Q/K/V biases)	SGLang	1443 · 1.4067	1446 · 1.4067 · 0.014	1442 · 1.4081 · 0.021
Llama-3.2-1B-Instruct (RoPE scaling, tied embeddings)	SGLang	1630 · 8.5347	1835 · 8.5313 · 0.043	1839 · 8.5007 · 0.061
Qwen3-1.7B-FP8 (128x128 blocks)	SGLang	1055 · 5.7789	1055 · 5.7752 · 0.064
Qwen3-1.7B-FP8 (128x128 blocks)	vLLM	605 · 5.7770	628 · 5.7815 · 0.058
Qwen3-30B-A3B-FP8 (MoE, 128 experts), TP=2	vLLM	359 · 1.3424	378 · 1.3428 · 0.033

Median TPOT of sealed and plain variants agreed within 0.5 ms in every row. In every measured keyed configuration, sealed output throughput at concurrency 16 was at most 0.3% below plain, and in some rows higher; the higher figures are attributed to noise on the shared host (the plain Llama run came first). Before weight locking was added, sealing took 14 to 20 s for the 1 to 2B models and 12 minutes for Qwen3-30B-A3B-FP8 on CPU.

Stopping servers

Each script stops the server processes it started, by their process IDs, when it exits. kllm serve replaces itself with the engine, so the recorded ID is the engine’s main process; whether its worker processes exit with it depends on the engine. h3_int8.sh starts each server in its own process group and stops that group. After an interrupted run, check nvidia-smi and stop any remaining server by its own process ID before the next run.