Supported models

kLLM serves two kinds of models. Plain serving runs any checkpoint the chosen engine supports, unchanged. Sealed serving runs a checkpoint that kllm seal has rewritten under a key; sealing is implemented for a fixed list of architectures, each with its own transforms and engines. This page lists them, together with what has been measured.

Plain serving

kllm serve MODEL without --key or --client launches SGLang or vLLM on the model path or Hugging Face id as given. kLLM adds no restriction of its own: any model the engine supports can be served this way, with the engine’s own arguments after --.

$ kllm serve MODEL
$ kllm serve MODEL --engine vllm
$ kllm serve MODEL --dry-run -- --context-length 32768

Built-in recipes apply to plain serving under SGLang only. Three recipes exist today, all for GLM-5.3 on H200 or B300; kllm recipes lists them. See Serving.

Sealed architectures

kllm seal selects the sealing scheme from the checkpoint: config.json for decoder LLMs, model_index.json for diffusion pipelines.

Family	Architecture	Transforms	Engines	Measured
Qwen3	`Qwen3ForCausalLM`	`orth`, `perm`	SGLang, vLLM	Qwen3-1.7B (SGLang, vLLM), Qwen3-8B (SGLang)
Qwen3-MoE	`Qwen3MoeForCausalLM`	`orth`, `perm`	SGLang, vLLM	FP8 checkpoint only (see below); bf16 covered by float64 tests
Qwen2 / Qwen2.5	`Qwen2ForCausalLM` (Q/K/V biases)	`orth`, `perm`	SGLang, vLLM	Qwen2.5-1.5B-Instruct (SGLang)
Llama 3.x	`LlamaForCausalLM` (Llama 3 RoPE scaling)	`orth`, `perm`	SGLang, vLLM	Llama-3.2-1B-Instruct (SGLang)
FP8 checkpoints of the above	`quant_method: fp8`, block-wise or per-tensor scales	`perm` only	SGLang, vLLM	Qwen3-1.7B-FP8 (SGLang, vLLM), Qwen3-30B-A3B-FP8 with TP=2 (vLLM)
MiniMax H3	FL2VA partition (`_class_name` containing `MiniMaxH3`)	`perm` (recommended), `orth`	SGLang (`sglang serve`)	FL2VA end to end under `perm`
Z-Image	`ZImagePipeline`, client-keyed scheme (`--client`)	`orth`, `perm`	kLLM latent server	`orth` against the plain diffusers pipeline

--transform defaults to orth. A sealed model records its transform, so kllm serve --key reads it from the sealed directory. Choosing a transform explains the trade-off; in short, perm keeps every weight a bit-exact rearrangement of an original one, and orth also hides per-channel statistics at the cost of rounding the rotated weights again.

Notes per family

Mixture of experts. The experts are permuted (router rows and expert tensors) and each expert’s MLP gets its own transforms.

FP8. perm moves whole quantization blocks and shuffles inside them, and the MLP’s powers of two go into the scales, so the FP8 values are only moved and sign-flipped and the sealed checkpoint dequantizes exactly to the transformed weights. orth would requantize and is refused. See Sealing FP8 and mixture-of-experts checkpoints.

MiniMax H3. The whole FL2VA chain is sealed: the Qwen3-VL text encoder, the DiT, the latent interfaces of the video and audio VAEs, and the tokenizer files. The DiT’s residual stream is always a signed permutation, because AdaLN scales channels one by one. H3 pipelines are served with SGLang only and cannot be served in client-keyed mode. perm is recommended, because orth re-rounds the rotated text-encoder weights, to which the audio stream is most sensitive. See Sealing and serving MiniMax H3.

Z-Image. A Z-Image pipeline is sealed by its own scheme. With --client it is sealed for client-keyed serving (Client-keyed mode): the text encoder under the LLM transforms and the DiT’s latent input and output in a key-derived patch basis. Only 2x2 image patching is supported. Without --client, kllm seal writes a separate demonstration scheme that ignores --transform; it is served only by kllm demo, which is under repair in 0.1.0.dev0, and kllm serve --key does not accept it.

Requirements checked at seal time

kllm seal refuses a checkpoint, with a message naming the reason, when:

the first entry of architectures is not one of the four LLM architectures above;
the model has MLP biases, or partial rotary embeddings (partial_rotary_factor other than 1.0);
the checkpoint uses a quantization method other than fp8, or is FP8 and --transform orth is requested;
the directory has no .safetensors files or no tokenizer.json;
the tokenizer’s regular vocabulary ids are not contiguous from 0, or an added token sits inside the regular vocabulary;
the checkpoint is already sealed;
for H3, the text encoder has attention biases; for Z-Image, the DiT uses a patching other than 2x2.

Measured results

All measurements were taken on NVIDIA A40 48 GB GPUs, one GPU per run except the TP=2 row, with SGLang 0.5.21 or vLLM 0.31.0, bf16, 1024 input and 128 output tokens, on a host shared with another tenant’s light job, as single runs without reported variance. Output throughput at concurrency 16, plain against sealed:

Model	Engine	Plain (tok/s)	`perm` (tok/s)	`orth` (tok/s)
Qwen3-1.7B	vLLM	1097	1094	1095
Qwen3-1.7B	SGLang	563	577	580
Qwen3-8B	SGLang	333	333	332
Qwen2.5-1.5B-Instruct	SGLang	1443	1446	1442
Llama-3.2-1B-Instruct	SGLang	1630	1835	1839
Qwen3-1.7B-FP8 (128x128 blocks)	SGLang	1055	1055	(not applicable)
Qwen3-1.7B-FP8 (128x128 blocks)	vLLM	605	628	(not applicable)
Qwen3-30B-A3B-FP8 (128 experts), TP=2	vLLM	359	378	(not applicable)

Sealed throughput matched plain serving in every measured keyed configuration, never more than 0.3% lower. The higher sealed figures in some rows are attributed to noise on the shared host; they are not a speed-up. Latency differences are visible in individual configurations, for example the median TTFT at concurrency 1 of Qwen3-1.7B under SGLang (98.0 ms plain, 105.2 ms perm). The A40 has no FP8 tensor cores, so both engines ran FP8 through weight-only kernels.

Quality of Qwen3-1.7B under vLLM, full test sets with lm-evaluation-harness (standard errors 0.004 for MMLU and 0.013 for GSM8K): MMLU 0.5560, 0.5562 and 0.5548, GSM8K strict match 0.6945, 0.6952 and 0.6960, for plain, perm and orth. All differences are within one standard error.

MiniMax H3 FL2VA sealed under perm ran as fast as plain end to end, for example 220.4 s for 8 steps in both cases. Before weight locking was added, sealing took 14 to 20 s for the 1 to 2B models, 12 minutes for Qwen3-30B-A3B-FP8 on CPU and about 15 minutes for H3 FL2VA (135 GB) on one A40. Benchmarking plain and sealed serving describes how to reproduce these measurements.

Request model support

Requests for new architectures are collected with the Model request issue form of the GitHub repository; Requesting model support describes the wishlist and kllm request. A request names the model, its architecture and the intended engine. To add an architecture directly, follow Adding a model family; every new family needs a float64 exactness test.