SCX: stateless KV-cache encoding

This page summarizes the SCX paper. Every result below is the paper’s own, reported under the paper’s conditions. kLLM 0.1.0.dev0 does not implement SCX, and no kLLM deployment profile has the properties described here. For the BibTeX entry, see Citing kLLM.

SCX: Stateless KV-Cache Encoding for Cloud-Scale Confidential Transformer Serving. Mu Yuan, Lan Zhang, Liekang Zeng, Siyang Jiang, Bufang Yang, Di Duan, Guoliang Xing. ACM SIGCOMM 2025. DOI 10.1145/3718958.3750509. Code: https://github.com/yuanmu97/scx.

The kLLM team publishes research on confidential transformer serving. The perm transform extends STIP’s permutation approach to grouped-query attention, RoPE, MoE and FP8; the research systems themselves are separate from what kllm seal ships today.

Important

Nothing on this page describes a feature of kLLM. SCX encodes activations and the KV-cache at run time, per request and per decoding step. kllm seal transforms the weights once, under one static key per sealed model. For what kLLM itself claims, read Claims.

Problem

A cloud that serves a transformer sees the user’s prompt and the generated text. The paper groups the existing remedies into three directions and states their costs:

Cryptographic protocols, such as multi-party computation and secret sharing. The paper cites the Puma protocol at about 200 s of latency and 1.8 GB of traffic for a 7B model, and notes quality loss from approximating non-linear layers.
CPU trusted execution environments (SGX, TDX). The paper measures a full-TEE deployment of a 7B model at 0.3 tokens/s.
Access control on special hardware (Apple Private Cloud Compute, NVIDIA confidential computing). Overhead is low on one device, but the paper points to the cost of inter-node encryption in distributed inference and to the dependence on specific GPU generations.

The stated goal is latency under 50 ms for 7B models, on ordinary hardware, while the user’s input and output remain private.

Approach

SCX encodes the KV-cache under keys that the user generates and controls. Its design goal is that, in the paper’s semi-honest model, the attention state held by the cloud cannot complete inference on its own. The paper calls this property a stateless KV-cache and defines it formally (Definition 2): the probability that the cloud produces the correct next token without the user’s key is bounded by a small value. The paper argues this bound analytically (Lemma 2).

The protocol has three stages.

Prefill. The user tokenizes and embeds the prompt locally. For each transformer block, the user sends an encoded input and the cloud computes the projections, attention and feed-forward layers on it. The encoding combines a token-level permutation, a feature-level permutation, redundant (decoy) embedding rows and, in the first block only, Laplace noise. The user removes the encoding between the cloud’s operations.
Key sharing. The user uploads the keys of the intermediate blocks and keeps the keys of the first and last blocks. The cloud recovers usable KV-caches for the intermediate blocks. This step reduces user-cloud interaction from O(T·L) to O(T+L), for T generated tokens and L layers. The paper recommends keeping one or two first blocks and one last block locked.
Generation. For each new token, the user encodes the token embedding with one-time keys. The cloud interacts with the user only at the first and last blocks. The user applies the classifier and samples the next token locally.

The user-side encoding and decoding can run in a CPU TEE on the cloud host, connected to the GPU over PCIe, in place of round trips to a user device. The paper states that SCX does not strictly depend on a TEE. It also describes the application of SCX to vision transformers, LLaVA and mixture-of-experts models such as Mixtral.

Threat model

Element	As stated in the paper
Parties	Two. The user is the data owner; the input and output must remain confidential. The cloud is the model owner; it hosts the proprietary model and performs the computation.
Adversary behavior	Both parties are semi-honest: they follow the protocol and try to infer information from the messages they observe. Active deviation by the cloud is not covered.
Data privacy goal	Recovering the user’s original input and output from the revealed information must be computationally infeasible for the cloud.
Model goal	The user must not gain full access to the proprietary model.
Out of scope	Retrieval-augmented generation providers and third-party tools; disaggregated prefill, decode and cache deployments (left to future work).

The difference from kLLM is fundamental. In SCX the cloud owns the model and knows the plaintext weights; SCX protects user data from the model owner. kLLM’s threat model is a deployer’s model, and its users’ data, against a cloud that does not own the model, within the measured limits stated in Claims. See Threat model.

Results reported in the paper

The paper evaluates six models from 1.1B to 70B parameters, including LLaVA-13B and Mixtral-8x7B, on WikiText-2, NarrativeQA and VQA. The cloud side used servers with four NVIDIA A100 GPUs and with four RTX 4090 GPUs; FP16 on A100 is the default setting.

Equivalence. Lemma 1 states that SCX decoding is mathematically equivalent to plaintext inference. In the experiments, SCX produced the same tokens as plaintext inference, with a perplexity difference of zero on WikiText-2. At FP16, the largest absolute logit difference reported is between 4.5×10^-5 and 7.2×10^-3.

Latency at an 8-token context. The paper’s headline figures use a context of 8 tokens. They are single-request figures and are not long-context or batched-serving results.

Time to first token and per-token latency, 8-token context (paper, Tables 6 and 7)
Method	TTFT, 7B	TTFT, 70B	One token, 7B	One token, 70B
Plaintext GPU	10 ms	112 ms	28 ms	202 ms
SCX	36 ms	324 ms	30 ms	208 ms
Full CPU TEE	17 s	182 s	4 s	26 s
Puma (MPC and HE)	200 s	not reported	164 s	not reported

Longer contexts. The overhead grows with the context. The paper reports prefill latency about ten times that of plaintext for contexts of 100 to 10,000 tokens, and more than 1,000 times faster than a full CPU TEE. Generation at a 10,000-token context took 9.3 s against 2.7 s for plaintext. Generating 1,000 tokens at a 100-token context took 330 s against 53 s.

Attacks. The paper evaluates a fitting attack (a two-layer surrogate model), a distance correlation test of the encoded embeddings, LLM-assisted reordering of shuffled tokens with GPT-4o, and a key-sharing attack in which the cloud reuses KV-caches of neighboring layers in place of the locked ones. In the last experiment, on Llama2-7B, token perplexity rose from 1.33 with the correct KV-cache to 19,108 on WikiText-2.

Limitations stated in the paper

The security analysis assumes two semi-honest parties.
Memory movement between the GPU and the CPU TEE dominates latency. For LLaVA-13B at a 1,000-token context, it accounted for 4,728 ms of a 5,051 ms prefill.
Communication grows with context length and model size.
Redundant embedding rows, needed for short inputs, add 13 to 25 ms to the time to first token (TinyLlama-1.1B, inputs of 10 to 100 tokens).
Low-precision formats introduce small numerical differences in the logits.
The privacy properties, differential privacy of the noised first-block embedding and the statelessness bound of Lemma 2, are argued analytically, not measured.

Relationship to kLLM

SCX and kLLM both rely on secret, exactly invertible linear transforms under which transformer computation commutes, so that the protected computation matches the plain one. They differ in what is transformed and for how long.

Aspect	kLLM `orth` and `perm` (implemented)	SCX (paper)
Who knows the plaintext weights	Not the cloud by design; it receives sealed weights. The transforms and Π of a public model, or of a fine-tune of one, are recoverable from them with the public checkpoint.	The cloud, which owns the model.
What is transformed	The weights, once, when the model is sealed.	Activations and the KV-cache, at run time.
Key lifetime	One static key per sealed model.	Per session in prefill; per decoding step in generation.
Token order and noise	Unchanged; no noise.	Shuffled in prefill, with decoy rows; Laplace noise in the first block.
Engine	Stock SGLang or vLLM, unchanged.	Per-block interception in a PyTorch and Hugging Face Transformers prototype.

Because kLLM’s transforms are static, its residual exposure differs from SCX’s. kLLM measures that exposure itself: against an operator who reads GPU memory, user text reduces to a substitution cipher over tokens, and the transforms and Π of a public model, or of a fine-tune of one, are recoverable from the sealed weights by a party that holds the public checkpoint. See Attacks and Exposure.