Adding an engine

This page describes how to add a launch backend, so that kllm serve --engine NAME starts another inference engine. kLLM 0.1.0.dev0 supports two engines, sglang and vllm, defined in src/kllm/backends/launch.py.

What an engine must provide

kLLM does not patch or wrap the engine. An engine can serve sealed models without changes if it meets these conditions.

  • It loads a Hugging Face checkpoint directory with its stock model class for the architecture. Sealed checkpoints keep the tensor names, shapes and dtypes of the original.

  • It accepts a tokenizer directory separate from the model directory, because the permuted tokenizer of a sealed LLM is decrypted into its own directory under /dev/shm.

  • For client-keyed mode, it starts without a tokenizer and takes and returns token ids over HTTP.

  • It is started as a single foreground process that kLLM can execute in place of itself.

How an engine is launched

cmd_serve in src/kllm/cli.py builds a LaunchSpec and passes it to exec_server(), which replaces the kllm process with the engine through os.execvpe. LaunchSpec.argv() holds all engine-specific logic:

Field

sglang

vllm

Base command

<python> -m sglang.launch_server --model-path MODEL

<bin>/vllm serve MODEL

Tensor parallelism

--tp N

--tensor-parallel-size N

Tokenizer

--tokenizer-path DIR

--tokenizer DIR

Diffusion pipelines

<bin>/sglang serve --model-path MODEL --num-gpus N

refused (ValueError)

Host and port

--host H --port P

--host H --port P

<python> is the running interpreter (sys.executable) and <bin> its directory: _tool(name) resolves a console script installed next to the interpreter, so the engine always comes from the same environment as kLLM. After the base command, argv() appends the recipe’s arguments and then the user’s passthrough arguments, so the user always has the last word.

When LaunchSpec.command is set (client-keyed Z-Image), argv() runs that command with --host, --port and the extra arguments, and no engine is involved.

Procedure

  1. Register the name. Add the engine to ENGINES in src/kllm/backends/launch.py. The --engine choices of kllm serve, kllm unlock and kllm client proxy are taken from this tuple.

    ENGINES = ("sglang", "vllm", "newengine")

  2. Render the command line. Add a branch to LaunchSpec.argv() that maps the model, the tensor-parallel size and spec.tokenizer onto the engine’s own flags. Keep the common tail unchanged: host and port, recipe arguments, then extra_args. If the engine cannot serve diffusion pipelines, raise ValueError when spec.diffusion is set, as the vLLM branch does.

  3. Check keyed serving. Provider.prepare_serve sets only spec.tokenizer for the l1 scheme, so no change is needed there if step 2 passes the tokenizer. MiniMax H3 pipelines are served with SGLang only, and prepare_serve refuses other engines for them.

  4. Support client-keyed mode. Provider.prepare_client_serve prepends --skip-tokenizer-init to the engine arguments of an LLM. If the new engine spells this option differently, handle it there. On the user’s side, src/kllm/keyed/client/llm.py defines one Engine subclass per engine (SGLang posts to /generate, VLLM to /v1/completions) and maps names to classes in its own ENGINES dictionary. Add a subclass whose run() sends token ids with the sampling parameters and stop ids, and yields Step objects with the new output ids, the finish reason and the usage counts. Engine-specific endpoints of the proxy are guarded by client.engine.name in src/kllm/keyed/client/proxy.py.

  5. Leave recipes alone, or add them separately. cmd_serve consults recipes only when --engine sglang is given, because every built-in recipe was measured on SGLang. Recipes for another engine need a change to that condition and their own measurements; see Contributing a serving recipe.

  6. Add tests. tests/test_cli.py shows the pattern: monkeypatch the GPU inventory, run cli.main([... "--dry-run"]) and assert on the printed command line. test_tokenizer_flag_per_engine checks the tokenizer flag of each engine directly on LaunchSpec:

    def test_newengine_flags():
    spec = LaunchSpec(model="m", tp=2, tokenizer="/dev/shm/t", engine="newengine")
    argv = spec.argv()
    assert argv[argv.index("--tokenizer") + 1] == "/dev/shm/t"

  7. Measure. Serve a supported model plain and sealed on the new engine with benchmarks/llm.sh and report the result in the pull request. For the client-keyed path, benchmarks/client.sh drives the proxy. See Benchmarking plain and sealed serving.

Checking the result

kllm serve --dry-run prints the command line without starting the engine:

$ kllm serve Qwen/Qwen3-8B --engine newengine --dry-run
$ kllm serve ./Qwen3-8B-sealed --key --engine newengine --dry-run -- --some-engine-flag

The second command unlocks the key and decrypts the tokenizer into /dev/shm before it prints the command line; remove the directory afterwards.

Note

benchmarks/llm.sh and benchmarks/client.sh know the engines of 0.1.0.dev0 only. A new engine needs its own branch in each script, for its server flags, its health endpoint and its load generator.

See also