Roadmap

This page lists the work planned for kLLM, roughly in order of priority within each section. It is maintained alongside ROADMAP.md in the repository. Each item is marked with its status:

  • Available in kLLM 0.1.0.dev0.

  • 0.2 line: present on the main branch, not yet in a tagged release; options may change before 0.2.0.

  • Planned: not available.

For what exists today, see kLLM at a glance and Command-line reference. To discuss an item or to work on one, open an issue in the issue tracker. Priorities and scope may change.

The 0.2 line: pull, run and keys

Version 0.2 is built around one workflow, in the manner of Ollama but on production engines: pull a model from a curated library, unlock it with a key and run it, with the same short model names in every step. The sealing and serving code of 0.1.0.dev0 is reused unchanged.

Item

Scope

Status

kllm pull, kllm list, kllm rm, kllm show, kllm search

A local model store with manifests, fed from a curated library of aliases such as qwen3:8b (Model library and naming).

0.2 line

kllm run, kllm ps, kllm stop, kllm logs

Background engine servers on the loopback interface, with an interactive chat session or a one-shot answer.

0.2 line

kllm key and kllm create

A named key ring, and sealing of a pulled model under a private key into the store.

0.2 line

kllm check

A manifest check of a local model; for an https:// address, the attestation binding check.

0.2 line

kllm verify, kllm cloud

The reproducibility check, which seals the public base again under the key and compares it tensor by tensor (Checking a published model); the keyless cloud part of a sealed model for client-keyed serving (kllm cloud).

0.2 line

Locked weights

kllm seal encrypts every weight in place under the key (--no-lock to skip); kllm serve --key and kllm cloud --key unlock them (Locked weights).

0.2 line

kllm doctor, kllm completion, kllm request

Host checks, shell completion and model requests (Requesting model support).

0.2 line

Official catalog

Public base models at pinned revisions, sealed under perm with locked weights, each under its own model key, with a kllm-manifest.json, published in the ANetResearch organization on Hugging Face (The catalog of official sealed models).

Rolling out

Model keys

One key per catalog model, derived from an offline master key and issued to approved applicants as passphrase-protected key files; every holder of a model’s key holds the same key (Model keys).

0.2 line

Website and key-application service

The public website and the service that reviews applications for model keys and delivers the issued key files (The kLLM website, The key-application service).

0.2 line

MiniMax H3 in the catalog

Held: the MiniMax H3 Community License excludes territories that a worldwide public repository cannot exclude; publication needs MiniMax’s authorization or a gated, territory-limited release (Licenses).

Planned

Artifact signing

Signatures on sealed artifacts, so that their origin can be checked before they are served. Manifests today record digests only.

Planned

Local runtime

Note

Planned — not available.

  • A resident local service on 127.0.0.1:11400 that serves every local model through one OpenAI-compatible endpoint and routes each request by its model field, starting plain models on demand.

  • An Ollama-compatible API: /api/chat and /api/generate, so that tools written for Ollama can use kLLM unchanged.

  • Keep-alive and admission control: unloading idle models after a configurable time, and admitting a model only when the GPUs have room for it.

  • A local web console for the local models, servers, keys and logs.

  • Modelfile-style recipes that name a model together with its prompt template, sampling parameters and engine arguments.

Until then, kllm run starts one background server per model, and kllm stop stops it.

Installation and distribution

Note

Planned — not available.

  • A published PyPI package, installable with pip install kllm, with the first tagged release. Today, install from a source checkout (Installation guide).

  • Installers for macOS and Windows.

  • Container images published for each SGLang and vLLM release. Today, build the image from the Dockerfile (Docker).

Security profiles

Profiles are trust boundaries, not scores (Deployment profiles).

Profile

Status

Official catalog

0.2 line, rolling out. Locked catalog models under per-model keys; every holder of a model’s key holds the same key, and the base models are public.

Private Seal

Available: kllm keygen and kllm seal, served with kllm serve --client. The key and the tokenizer stay on the user’s machine.

Attested CPU

Available with kLLM Enterprise: kllm serve --key-source cdh --production in a confidential VM. Not yet validated on TEE hardware.

Confidential GPU

Planned. See Confidential computing below.

Models

Note

Planned — not available.

  • DeepSeek-V3 and Kimi K2 (multi-head latent attention, shared experts) and GLM-4.5 and later.

  • Shared experts in the style of Qwen2-MoE.

  • Weight-only integer quantization (AWQ, GPTQ), with permutations restricted to whole quantization groups, as FP8 blocks are today.

  • The Gemma and Mistral families.

  • Further models requested on the wishlist (Requesting model support).

The architectures that can be sealed today are listed in Supported models.

Engines and deployment

Note

Planned — not available.

  • Tensor and expert parallelism measured on multi-GPU hosts for every supported family.

  • NVIDIA Dynamo and llm-d setups run end to end and kept in continuous integration. The current examples have not been run end to end; see NVIDIA Dynamo and Kubernetes and llm-d.

Evaluation

Note

Planned — not available.

  • lm-eval-harness results (MMLU, GSM8K, wikitext) for plain and sealed models, next to the perplexity table.

  • GPU continuous integration on a self-hosted runner for every release. Today, the GPU workflow runs on demand only.

Confidential computing

Note

Planned — not available.

  • Validation on Intel TDX, AMD SEV-SNP and Hygon CSV hosts with GPU passthrough, with measurements of the CPU-side cost.

  • NVIDIA confidential computing (H100, H200 and B200 in CC mode) for deployments that also need GPU memory protected. This is the Confidential GPU profile.

  • A design target for disaggregated serving: combined with confidential GPUs, tensors that are already transformed under the key could cross nodes without an additional encryption step. This is a design target, not a measured property.

Until then, no kLLM profile protects GPU memory, and a confidential VM with an ordinary GPU does not provide the protection of a confidential GPU. See Claims.