Contributing to kLLM

Issues and pull requests are welcome. For anything larger than a fix, open an issue first so that the approach can be agreed before the work starts. This page is maintained alongside CONTRIBUTING.md in the repository.

There are several ways to contribute:

  • report a defect or ask a question in the issue tracker;

  • request support for a model architecture by opening an issue that names the architecture (the architectures entry of its config.json) and a public checkpoint;

  • contribute code, a model family, a serving recipe or a launch backend;

  • contribute measurements made with the scripts in benchmarks/;

  • improve the documentation.

Note

Requests for new models use the Model request issue form; see Requesting model support.

Reporting issues

A useful report states:

  • the output of kllm version;

  • the command line, with the engine and its version (SGLang or vLLM);

  • the model architecture and, for a sealed model, the transform used;

  • the GPU model and count, as reported by nvidia-smi;

  • the complete error message.

kllm serve --dry-run prints the engine command without starting it, which often helps to narrow down a launch problem.

Warning

Never attach a key file, a passphrase, a value of KLLM_MODEL_KEY, a trusted.kllm file or a decrypted tokenizer to an issue. A sealed checkpoint together with its key is the plain model. A key fingerprint, as printed by kllm keygen, is safe to share.

Development setup

Clone the repository and install the dev extra in a virtual environment:

$ git clone https://github.com/ANetResearch/kLLM
$ cd kLLM
$ uv venv --python 3.11 && uv pip install -e '.[dev]'
$ .venv/bin/pytest && .venv/bin/ruff check .

The unit tests run on CPU in float64 and need neither a GPU nor a model download. Continuous integration runs ruff check . and pytest -q on Python 3.10 and 3.12 for every push and pull request, and checks that the package builds. On a machine without a GPU, the CPU build of PyTorch keeps the installation small; continuous integration installs it as follows:

$ uv pip install -e '.[dev]' --index-strategy unsafe-best-match \
    --extra-index-url https://download.pytorch.org/whl/cpu

The dev extra does not include the dependencies of client-keyed mode and Z-Image, so their tests are skipped. To run them as well, install the client and image extras and httpx, which the FastAPI test client needs:

$ uv pip install -e '.[dev,client,image]' httpx

Tests that need a CUDA GPU, such as those of the INT8 kernels, are skipped on CPU. The end-to-end GPU workflow (.github/workflows/gpu.yml) seals a model under perm, serves it with SGLang and vLLM, and compares it with plain serving through benchmarks/llm.sh. It runs on demand on a self-hosted runner.

What a change needs

Exactness

A sealed model must compute the plain model’s outputs. Every transform is covered by a float64 reference forward pass (tests/test_schemes.py, tests/test_families.py, tests/test_h3.py) that compares sealed and plain logits to 1e-9. A new model family or tensor kind comes with such a test.

Speed

The sealed checkpoint keeps every tensor’s name, shape and dtype, so the engine runs its stock model class and kernels. A change that needs engine patches or extra work per token is out of scope.

Measurements for claims

Numbers in the README come from benchmarks/. If a change moves them, rerun the script and update the table with the hardware and the versions used. State every number with its conditions: GPU, engine and version, precision, input and output lengths and concurrency.

Security claims

The security documentation states what an attacker learns, with the attack that measures it in tools/attacks/. Keep claims to what is measured. See Claims and Attacks.

Code style

ruff check . must pass. The line length is 100 characters and the target is Python 3.10.

Adding a model family

  1. Map the checkpoint’s tensors onto the transforms in src/kllm/keyed/schemes/transforms.py (Transforms._layer_tensor).

  2. Add the architecture to SUPPORTED_BASES in src/kllm/keyed/schemes/base.py, and reject the configuration options that you do not handle in check_config (src/kllm/keyed/schemes/seal.py).

  3. Extend the reference forward pass in tests/test_families.py and check exactness under both transforms.

  4. Serve a real checkpoint plain and sealed with benchmarks/llm.sh and report the result in the pull request:

    $ benchmarks/llm.sh ~/models/Qwen3-8B
$ ENGINE=vllm benchmarks/llm.sh ~/models/Qwen3-8B
$ python benchmarks/report.py out/Qwen3-8B-sglang

The procedure is described in detail in Adding a model family. Serving recipes, launch backends and key sources are covered in Contributing a serving recipe, Adding an engine and Registering a key source.

Documentation

The documentation is written in reStructuredText and built with Sphinx and the Read the Docs theme. Build it locally with warnings treated as errors, as continuous integration does:

$ pip install -r docs/requirements.txt
$ python -m sphinx -n -W --keep-going -b html docs docs/_build/html

The published documentation is built the same way; see Documentation hosting.

Documentation copy is formal and precise. State numbers with units and conditions, and label planned features as planned.

License of contributions

kLLM is licensed under Apache-2.0. By contributing, you agree that your contribution is licensed under the same terms, as provided in section 5 of the license. See License.