Sealing FP8 and mixture-of-experts checkpoints

kllm seal accepts mixture-of-experts (MoE) checkpoints of the Qwen3-MoE family and FP8 checkpoints of every supported LLM family. This page lists the requirements, gives the procedure, and reports what was measured. Sealing describes the transforms in general.

Supported checkpoints

Checkpoint	Transforms	Requirements
Qwen3-MoE (`Qwen3MoeForCausalLM`), bf16	`orth`, `perm`	As for dense models: safetensors weights, a `tokenizer.json` with contiguous regular ids, no MLP biases, no partial rotary embeddings.
FP8 checkpoints of Qwen3, Qwen3-MoE, Qwen2 / Qwen2.5 and Llama 3.x	`perm` only	`quantization_config.quant_method` is `fp8`; block-wise or per-tensor weight scales stored as `weight_scale` or `weight_scale_inv`; square blocks whose size equals the head dimension.

Other quantization methods are refused, for example awq checkpoints are not supported (FP8 is).

Note

Planned — not available in 0.1.0.dev0. AWQ and GPTQ checkpoints, MoE models with shared experts (Qwen2-MoE), and the DeepSeek-V3 and Kimi K2 families. See Roadmap.

How MoE models are sealed

In each MoE layer the experts are permuted: the rows of the router and the expert tensors move to new expert slots together, so routing selects the same expert under its new index. Each expert’s MLP is then transformed like a dense MLP, under its own permutation of the intermediate dimension and its own ±2^k scales. Dense MLP layers of an MoE model (mlp_only_layers) are transformed as in a dense model. Tensor names and shapes are unchanged, so the engine loads the sealed checkpoint with its own MoE implementation and expert-parallel options.

How FP8 checkpoints are sealed

FP8 weights are stored with one scale per block (block-wise) or one scale per tensor. To keep each FP8 value exact, the perm sealer structures every permutation around the blocks:

whole quantization blocks move, and the elements of a block are shuffled inside it;
the MLP’s powers of two are constant per block and are moved into the scales; for per-tensor scales they are 1;
the FP8 values themselves are only moved and sign-flipped, so the sealed checkpoint dequantizes exactly to the transformed weights;
each scale keeps its dtype and shape.

orth would mix values across blocks and require requantization, which would change the model; it is therefore refused.

Procedure

Download the checkpoint as a local directory. kllm seal does not accept Hub ids.

$ hf download Qwen/Qwen3-30B-A3B-FP8 --local-dir ~/models/Qwen3-30B-A3B-FP8

Seal it under perm. Sealing runs on a CUDA GPU when one is available and on the CPU otherwise; Qwen3-30B-A3B-FP8 took 12 minutes on CPU, measured before weight locking was added.
```
$ kllm seal ~/models/Qwen3-30B-A3B-FP8 -o ~/models/Qwen3-30B-A3B-FP8-sealed --transform perm
```
Serve it. The measured configuration used vLLM with tensor parallelism 2:
```
$ kllm serve ~/models/Qwen3-30B-A3B-FP8-sealed --key --engine vllm --tp 2
```
SGLang is also supported for FP8 checkpoints. MoE models under SGLang have not been measured.
Check the result against plain serving with benchmarks/llm.sh and TRANSFORMS=perm (see Benchmarking plain and sealed serving).

Errors specific to FP8

Message	Cause
`FP8 checkpoints need --transform perm (orth would requantize)`	`--transform orth` (the default) on an FP8 checkpoint. Add `--transform perm`.
`<tensor> is FP8 but has no weight_scale(_inv)`	An FP8 weight without a matching scale tensor.
`FP8 blocks must be square, got <block>`	`weight_block_size` is not square.
`FP8 blocks of <n> need head_dim <n>`	The block size differs from the head dimension.
`<n> is not a multiple of the quantization block <block>`	A permuted dimension cannot be split into whole blocks.
`<tensor>: the transform does not respect the quantization blocks`	The sealer’s own check failed; the checkpoint layout is not supported.

Measured results

One NVIDIA A40 48 GB (two for TP=2), SGLang 0.5.21 or vLLM 0.31.0, 1024 input and 128 output tokens, GPU shared with another tenant’s light job, single runs. Each cell gives output throughput at concurrency 16, perplexity over 14690 tokens of license texts, and for the sealed model the mean |Δ log-prob| against plain.

Model	Engine	plain	`perm`
Qwen3-1.7B-FP8 (128x128 blocks)	SGLang	1055 tok/s · 5.7789	1055 tok/s · 5.7752 · 0.064
Qwen3-1.7B-FP8 (128x128 blocks)	vLLM	605 tok/s · 5.7770	628 tok/s · 5.7815 · 0.058
Qwen3-30B-A3B-FP8 (MoE, 128 experts), TP=2	vLLM	359 tok/s · 1.3424	378 tok/s · 1.3428 · 0.033

The higher sealed figures are attributed to noise on the shared host. The A40 has no FP8 tensor cores, so both engines run FP8 through weight-only Marlin kernels; the sealed FP8 values are the original bytes, moved and sign-flipped. The test suite checks MoE exactness under both transforms and FP8 exactness under perm for block-wise and per-tensor scales, in float64 to 1e-9 on small random models.

Security notes

The general limits of static transforms apply; see Claims.

A cloud that reads the sealed FP8 weights sees which channels share a quantization block, but not the order of the blocks or the order inside a block.
MoE and FP8 checkpoints are sealed under perm in the measured configurations, and a public model, or a fine-tune of one, is recoverable under perm by a party that holds the public checkpoint. The column-profile attack on the LM head applies unchanged. Matching experts to those of the base model by their statistics is expected to work as for MLP channels; this has not been measured on these models.