Contributing a serving recipe

A recipe is a set of SGLang arguments measured for one model on one GPU family. When the model and the local GPUs match a recipe, kllm serve applies its arguments and its tensor-parallel size, so that the measured configuration is reproduced without further flags. This page describes the recipe format, how recipes are matched and what a new recipe needs. For the user’s view, see Serving.

Built-in recipes

kLLM 0.1.0.dev0 ships three recipes, all for GLM-5.3 and all measured on SGLang. List them with kllm recipes.

Name	Model pattern	GPU families	Tensor-parallel size
`glm-5.3-h200`	`glm-5.3`	`H200`	8
`glm-5.3-b300`	`glm-5.3`	`B300`	8
`glm-5.3-nvfp4-b300`	`glm-5.3-nvfp4`	`B300`	8

Models without a recipe run with the engine’s defaults.

The recipe format

A recipe is a frozen dataclass, Recipe, in src/kllm/recipes.py:

Field	Type	Meaning
`name`	`str`	Unique name, used by `kllm serve --recipe NAME`. By convention `<model>-<gpu>` in lower case.
`models`	`tuple[str, ...]`	`fnmatch` patterns matched against the model key (see below).
`gpu_families`	`tuple[str, ...]`	GPU families the recipe was measured on.
`tp`	`int`	Tensor-parallel size; used unless the user passes `--tp`.
`args`	`tuple[str, ...]`	SGLang arguments, one token per element.
`summary`	`str`	One line printed by `kllm recipes`.
`source`	`str`	Reference to the report that holds the measurements.
`env`	`dict[str, str]`	Environment variables for the engine (default empty).

Matching

recipes.find(model, gpu_family, name) selects the recipe:

If name is given (--recipe NAME), the recipe with that name is returned, whatever the model and the GPUs. An unknown name raises KeyError, which kllm reports with exit status 2.
Otherwise the model argument is reduced to its model key: trailing slashes removed, last path component, lower case. /models/GLM-5.3/ and zai-org/GLM-5.3 both become glm-5.3.
The first recipe in BUILTIN whose models patterns match the model key, and whose gpu_families contain the detected GPU family, is returned. If no GPU is detected, or the first GPU belongs to no known family, the family is not checked.

Because the first match wins, a more specific recipe must come before a more general one whose pattern also matches it. Recipes are consulted only when kllm serve runs with --engine sglang and without --no-recipe.

The detected GPU family is the family of the first visible GPU, derived from its nvidia-smi name in src/kllm/gpu.py:

Substring of the GPU name	Family
`B300`	`B300`
`B200`	`B200`
`H200`	`H200`
`H100`	`H100`
`H20`	`H20`
`A100`	`A100`
`A40`	`A40`
`RTX PRO 6000`	`RTXPRO6000`
`L40S`	`L40S`

The substrings are checked in this order. A GPU that matches none has no family; as on a host without GPUs, the family is then not checked, so a recipe whose model pattern matches is applied with its tensor-parallel size. Pass --no-recipe on such hosts if the recipe does not fit. To support a new GPU, add an entry to _FAMILIES before any entry whose substring it also contains.

How a recipe is applied

The engine command line is assembled in this order:

kLLM’s base command: model path, tensor-parallel size and, for a sealed model, the tokenizer path;
--host and --port;
the recipe’s args;
the arguments after -- on the kllm serve command line.

Passthrough arguments therefore override recipe arguments where SGLang takes the last occurrence of a flag. The recipe’s env is merged into the engine environment before the variables that kLLM sets itself. Inspect the result with kllm serve MODEL --dry-run.

Requirements for a new recipe

Measured: Every argument of the recipe has been measured on the stated GPU family with the stated tensor-parallel size. State the workload (input and output lengths, concurrency), the SGLang version, the number of runs and the result against SGLang’s defaults on the same host in the pull request. source points at that report.
Scoped: The models patterns match only the checkpoints that were measured. A recipe for one quantization of a model must not match another quantization; glm-5.3 and glm-5.3-nvfp4 are kept apart for that reason.
Engine arguments only: A recipe consists of SGLang arguments and environment variables. It must not depend on patches to SGLang or on files outside the model directory.
Independent of sealing: A recipe applies to plain and sealed serving alike. Because sealed checkpoints keep the tensor names, shapes and dtypes of the original, a recipe measured on the plain model applies to the sealed one; report a sealed run as well if the recipe changes kernels that read the weights.

Procedure

Add a Recipe to BUILTIN in src/kllm/recipes.py. Reuse the shared argument groups of the module where they apply, and keep the arguments one token per element:

Recipe(
    name="model-x-h100",
    models=("model-x",),
    gpu_families=("H100",),
    tp=4,
    args=(
        "--mem-fraction-static", "0.85",
        "--chunked-prefill-size", "8192",
    ),
    summary="one line that states what the recipe changes",
    source="reference to the measurement report",
),

Add tests to tests/test_recipes.py: the recipe is found by Hub identifier and by local path, it is not found for another GPU family, and it does not shadow or get shadowed by an existing recipe. test_recipe_names_unique checks that names stay unique.

Check the command line with a dry run on a matching host, or with a monkeypatched GPU inventory as in tests/test_cli.py:

$ kllm recipes
$ kllm serve /models/Model-X --dry-run
$ kllm serve /models/Model-X --recipe model-x-h100 --dry-run

Open a pull request with the measurement report.

Note

Recipes for vLLM are not supported in 0.1.0.dev0: kllm serve --engine vllm always uses vLLM’s defaults. See Adding an engine.