Logo

First steps

  • kLLM at a glance
    • Where kLLM sits
    • Three properties
    • Deployment profiles
    • Supported models
    • What kLLM is not
    • Research
    • Licensing
    • Next steps
  • Installation guide
    • Requirements
    • Installing from source
      • Extras
    • Verifying the installation
      • Commands of the 0.2 line
    • Installing with Docker
    • Development installation
    • Next steps
  • Quickstart
    • Run an official sealed model with its model key
      • Step 1: apply for model keys
      • Step 2: import the key
      • Step 3: run the model
      • Step 4: use the server
    • Seal a model under a private key
      • Step 1: create a key
      • Step 2: download and seal
      • Step 3: run the sealed model
    • What the key protects
    • Next steps
  • Examples
    • Keyed serving
    • Client-keyed serving
    • MiniMax H3
    • The two-stage demo

Basic concepts

  • Concepts
    • The workflow at a glance
    • Terms
      • Model key
      • Key file and passphrase
      • Fingerprint
      • Sealed checkpoint
      • Transform: orth and perm
      • Vocabulary permutation
      • Permuted tokenizer and trusted.kllm
      • Trusted domain
      • Engine
      • Launcher
      • Recipe
      • Key source
      • Client-keyed mode
      • Deployment profile
    • What sealing does not hide
    • See also
  • Model library and naming
    • Model references
    • Aliases and tags
    • Library contents
    • Official sealed models
    • Local store
    • Store record and manifests
    • See also
  • Model keys
    • What a key is
    • Creating a key
    • Key file format
    • Fingerprints
    • Passphrases
      • Supplying the passphrase
    • Backups
    • Changing the passphrase
    • Rotating a key
    • Keys in a confidential VM
    • Key ring
    • See also
  • The catalog of official sealed models
    • Models
    • What a published model contains
    • Keys
    • Running a catalog model
    • Checking a published model
    • Licenses
    • Release timing
    • See also
  • Model keys
    • What a model key is
    • What a model key is not
    • Applying for model keys
    • Using a model key
    • Limits
    • See also
  • Sealing
    • Synopsis
    • Supported checkpoints
      • Requirements for a language-model checkpoint
    • Transforms
      • The residual stream
      • Shared by both transforms
      • orth and perm compared
      • Exactness
    • FP8 checkpoints
    • Pipelines
      • MiniMax H3
      • Z-Image
    • Files written
    • Locked weights
    • Sealing time
    • Measured effect on serving
    • See also
  • Serving
    • Synopsis
    • Plain serving
    • Sealed serving
      • Locked weights
    • Engines
    • Tensor parallelism and GPUs
    • Recipes
    • Engine arguments
    • Dry runs
    • Host, port and process
    • Client-keyed serving
    • Production mode and key sources
    • Other launchers: kllm unlock
    • See also
  • Client-keyed mode
    • Overview
    • Sealing for the cloud
    • Serving in the cloud
    • The local proxy (LLMs)
    • Z-Image
    • Measured results
    • What this mode protects
    • See also
  • Supported models
    • Plain serving
    • Sealed architectures
    • Notes per family
    • Requirements checked at seal time
    • Measured results
    • Request model support
    • See also
  • Requesting model support
    • The wishlist
    • Opening a request
    • Contributing support
    • See also
  • Deployment profiles
    • Summary
    • Official catalog
    • Private Seal
    • Attested CPU
    • Confidential GPU
    • Choosing a profile
    • See also
  • Sealed checkpoint format
    • How kLLM recognizes a sealed directory
    • Decoder LLMs
    • The trusted.kllm blob
    • MiniMax H3 pipelines
    • Z-Image pipelines
    • Locked weights
    • What the format protects
    • Compatibility
    • Planned
    • See also

Security

  • Threat model
    • Parties
    • Profiles and adversaries
    • Attested CPU: serving in a confidential VM
    • Private Seal: client-keyed serving without a confidential VM
    • Development mode: kllm serve --key outside a confidential VM
    • Storage thief
    • Outside the model in every profile
    • See also
  • Exposure
    • Serving in a confidential VM
      • The transforms
      • Exactness
    • Locked weights
    • Client-keyed serving
      • What the Z-Image client seal changes
      • Measured log check
    • Where keys and decrypted tokenizers live
    • See also
  • Attacks
    • Weights of a public model, or of a fine-tune of one
      • Under orth
      • Under perm
      • Interpretation
    • User tokens
    • MiniMax H3 outputs
    • A stolen checkpoint, or a wrong key
      • Locked weights
    • Client-keyed Z-Image latents
    • See also
  • Claims
    • Design targets
    • Claims by profile
    • Not protected in client-keyed mode
      • Token ids are a substitution cipher
      • Z-Image latents in GPU memory show the image
      • The serving process sees everything it computes
      • Transport and access control are not provided
    • Development mode is not a profile
    • Performance and fidelity statements
    • What kLLM claims
    • What kLLM does not claim
    • See also

Deployment

  • Docker
    • Building the image
    • Contents of the image
    • Running a plain model
    • Creating a key and sealing in a container
    • Serving a sealed model with a key
    • Serving in client-keyed mode
    • See also
  • Confidential VM deployment
    • Components
    • Prerequisites
    • Procedure
      • 1. Seal the model on the deployer’s machine
      • 2. Store the key in the key broker
      • 3. Start the server in the VM
      • 4. Start the attesting gateway (kLLM Enterprise)
      • 5. Publish reference values
    • Production mode
    • Running without hardware
    • Limitations
    • See also
  • Verifying a deployment
    • Report data
    • The /attestation endpoint
    • Running the binding check
      • How the binding is checked
    • Verifying the quote
    • Using the helper from Python
    • See also
  • NVIDIA Dynamo
    • The kllm unlock command
    • Launching the workers
    • Placing components in the trusted domain
    • Cleaning up
    • See also
  • Kubernetes and llm-d
    • Prerequisites
    • Deploying the example
    • Running SGLang instead of vLLM
    • llm-d
    • See also
  • The kLLM website
    • Repository layout
    • Building
    • Deploying the public site
    • Deploying the administration console
    • Production deployment
    • Content maintenance
    • See also
  • The key-application service
    • Interfaces
    • Security properties
    • Configuration
    • Outgoing mail
    • Deployment
    • Issuing model keys
    • Administrator accounts
    • Reviewing applications
    • Operations
    • See also
  • Documentation hosting
    • Address layout
    • Building
    • Publishing
    • See also

Solving specific problems

  • Frequently asked questions
    • Is sealing a form of encryption?
    • Does sealing slow down inference?
    • Does sealing change the model’s outputs?
    • Which models can be sealed?
    • Which inference engine should be used?
    • Does kLLM fork or patch SGLang or vLLM?
    • What does the cloud see?
    • Is kllm serve --key safe on an ordinary cloud host?
    • Can someone with the sealed files but without the key run the model?
    • Does sealing protect the weights of a fine-tuned model?
    • How is the key of an official sealed model obtained?
    • How can an official sealed model be checked against its public base?
    • Should orth or perm be used?
    • What happens if the key file or its passphrase is lost?
    • What is kLLM Enterprise?
    • How does kLLM relate to Ollama?
    • How does kLLM relate to SCX and STIP?
    • Does kLLM work with Dynamo, llm-d and Kubernetes?
    • Are confidential GPUs supported?
  • Seal and serve a checkpoint by hand
    • Step 1: download a checkpoint
    • Step 2: create a model key
    • Step 3: seal the checkpoint
    • Step 4: serve the sealed model
    • Step 5: send a request
    • Running the whole sequence as a script
    • What the key protects in this setup
    • Next steps
  • Choosing a transform
    • Quick decision
    • Comparison
    • Fidelity
    • Exposure
    • Comparing both on a model
    • See also
  • Serving with vLLM
    • Installation
    • Serving a model
    • Client-keyed serving
    • Other launchers
    • Not available with vLLM
    • Measured results
    • See also
  • Sealing FP8 and mixture-of-experts checkpoints
    • Supported checkpoints
    • How MoE models are sealed
    • How FP8 checkpoints are sealed
    • Procedure
    • Errors specific to FP8
    • Measured results
    • Security notes
    • See also
  • Sealing and serving MiniMax H3
    • Requirements
    • Procedure
    • What kllm serve does
    • What is sealed
    • Results
    • Security notes
    • See also
  • Running the H3 diffusion transformer on INT8 kernels
    • Requirements
    • Usage
      • With a sealed H3 pipeline
      • With plain H3
    • How it works
    • Interaction with sealing
    • Measured results
    • Reproducing
    • See also
  • Generating images in client-keyed mode
    • Roles and requirements
    • Procedure
    • What crosses the network
    • Measured results
    • What this protects
    • Z-Image without --client
    • See also
  • Benchmarking plain and sealed serving
    • Measurement conditions
    • Prerequisites
    • LLM latency, throughput and perplexity
    • Standard evaluations
    • Client-keyed serving
    • MiniMax H3
    • Attacks
    • Which script produced which result
    • Results: SGLang
    • Results: other families
    • Stopping servers
    • See also
  • Troubleshooting
    • General diagnosis
    • Installation and extras
    • Keys and passphrases
    • Sealing
    • Serving
    • Client-keyed mode
    • Engine errors and output problems
    • See also

Extending kLLM

  • Architecture overview
    • Source layout
    • The provider layer
    • The scheme layer
    • The launch layer
    • The request path of kllm serve
    • Extension points
    • See also
  • Adding a model family
    • Requirements
    • How tensors are classified
    • Procedure
    • Tokenizer requirements
    • Security review
    • See also
  • Adding an engine
    • What an engine must provide
    • How an engine is launched
    • Procedure
    • Checking the result
    • See also
  • Contributing a serving recipe
    • Built-in recipes
    • The recipe format
    • Matching
    • How a recipe is applied
    • Requirements for a new recipe
    • Procedure
    • See also
  • Registering a key source
    • Built-in and registered sources
    • The interface
    • Where key sources are called
    • Registering a key source
    • Security requirements
    • See also

Reference

  • Command-line reference
    • General behavior
    • kllm version
    • kllm recipes
    • kllm keygen
    • kllm seal
    • kllm serve
    • kllm unlock
    • kllm verify
    • kllm cloud
    • kllm client
      • kllm client proxy
      • kllm client generate
    • kllm demo
      • kllm demo up
      • kllm demo run
      • kllm demo down
    • Model, key and runtime commands (0.2 line)
      • kllm pull
      • kllm run
      • kllm list
      • kllm ps
      • kllm stop
      • kllm rm
      • kllm show
      • kllm check
      • kllm create
      • kllm key
      • kllm doctor
      • kllm search
      • kllm request
      • kllm logs
      • kllm completion
      • Planned commands
    • Module entry points
      • python -m kllm.keyed.attest
      • python -m kllm.kernels.h3
      • python -m kllm.keyed.client.server
  • Settings
    • Overview
    • kLLM variables
      • KLLM_PASSPHRASE
      • KLLM_UNLOCKED_DIR
      • KLLM_HOME
      • KLLM_LIBRARY
      • KLLM_PRODUCTION
      • KLLM_KEY_SOURCE
      • KLLM_MODEL_KEY
      • KLLM_MODEL_DIR
      • KLLM_TRUSTED_FROM
      • KLLM_INT8
    • kLLM Enterprise variables
    • Other variables
    • Tool and benchmark variables
    • Files and paths
  • Glossary

Research

  • SCX: stateless KV-cache encoding
    • Problem
    • Approach
    • Threat model
    • Results reported in the paper
    • Limitations stated in the paper
    • Relationship to kLLM
    • See also
  • STIP: three-party privacy-preserving inference
    • Problem
    • Threat model
    • Approach
    • Results reported in the paper
    • Limitations stated in the paper
    • Relationship to kLLM
    • See also
  • Citing kLLM
    • Cite kLLM
    • Cite the research
      • SCX
      • STIP
    • Which work to cite
    • Models and code

All the rest

  • Release notes
    • Upcoming 0.2 line (main branch)
      • New features
      • Changes
      • Known issues
    • Unreleased (0.1.0.dev0)
      • Highlights
      • New features
      • Changes
      • Known issues
    • 0.1.0
  • Roadmap
    • The 0.2 line: pull, run and keys
    • Local runtime
    • Installation and distribution
    • Security profiles
    • Models
    • Engines and deployment
    • Evaluation
    • Confidential computing
  • kLLM Enterprise
    • Offering
    • What Enterprise does not change
    • Contact
  • Contributing to kLLM
    • Reporting issues
    • Development setup
    • What a change needs
    • Adding a model family
    • Documentation
    • License of contributions
  • Versioning and format stability
    • Version numbers
    • Pre-1.0 status
    • Sealed-format compatibility
  • License
    • Open-source core
    • kLLM Enterprise
    • Third-party code and dependencies
    • Contributions
kLLM
  • Search

© Copyright Agent Network.

Built with Sphinx using a theme provided by Read the Docs.