STIP: three-party privacy-preserving inference

This page summarizes the STIP paper. Every result below is the authors’ own, reported under the paper’s conditions. kLLM does not inherit STIP’s security properties; the section Relationship to kLLM states where the two coincide and where they differ. For the BibTeX entry, see Citing kLLM.

STIP: Three-Party Privacy-Preserving and Lossless Inference for Large Transformers in Production. Mu Yuan, Lan Zhang, Yihang Cheng, Miao-Hui Song, Guoliang Xing, Xiang-Yang Li. Network and Distributed System Security (NDSS) Symposium 2026, San Diego. DOI 10.14722/ndss.2026.230035. The paper lists its code at https://github.com/yuanmu97/secure-transformer-inference.

The kLLM team publishes research on confidential transformer serving. The perm transform extends STIP’s permutation approach to grouped-query attention, RoPE, MoE and FP8; the research systems themselves are separate from what kllm seal ships today.

Important

Nothing on this page is a claim about kLLM. In particular, kLLM’s threat model, its permuted-token interface and its single static key per sealed model differ from STIP. For what kLLM itself claims, read Claims.

Problem

A transformer service deployed in a public cloud exposes the user’s data to the cloud. Deployed on the user’s device, it exposes the model’s parameters to the user. Splitting the model between the two leaks information through activation inversion. Protocols that combine two-party computation with other cryptographic primitives give cryptographic guarantees, but the paper describes them as lossy, because non-linear layers are approximated, and as far too slow and communication-heavy for large models.

The authors set four design goals: privacy of both data and parameters; no accuracy loss, with no approximation of any computation; compatibility with production frameworks, including the KV cache; and extension to transformer variants.

Threat model

STIP separates three parties.

Party	Role
P1, model developer	Trains and owns the private parameters. Generates the permutations and transforms the model.
P2, model server	Owns the compute, for example GPUs in a public cloud. Runs the transformed model.
P3, data owner	Owns the private input and the inference output.

The parties are semi-honest: each follows the protocol and tries to infer information from what it observes. The security goals are that P1 and P2 learn neither the input nor the output of P3, and that the parameters are hidden from P2 and P3.

Warning

STIP assumes that the model developer and the model server do not collude. The authors state that collusion “reverts the system to a standard two-party setting”, in which the privacy of the data owner’s input may be compromised. The paper also discusses, informally, a server that poses as a user to obtain the shared permutations, and proposes several model instances with distinct transformations as a mitigation.

Approach

Feature-space permutation of the parameters. The model developer draws a secret permutation of the residual stream, three permutations per layer and a permutation of the classifier output. It rewrites every weight matrix and every normalization parameter with them. The paper proves (Theorem 1) that the transformed model, applied to a permuted input, produces the permuted output of the original model. The proof relies on the permutation equivariance of element-wise activations, softmax and layer normalization. The paper explains why it permutes features rather than sequence positions: sequence permutation conflicts with the causal mask of decoders.

Semi-symmetric sharing. Only the input and output permutations are shared with users. The per-layer permutations stay with the model developer.

Workflow.

The developer generates the permutations, transforms the model and uploads it to the server. It distributes the input and output permutations to registered users.
The user tokenizes and embeds the input on the device, permutes the embedding and sends it to the cloud.
The cloud runs the transformed model with unchanged inference code and returns permuted logits. The user reverses the output permutation and samples locally.

Because the cloud switches to a different set of weights without changing the inference code, the KV cache and production frameworks continue to work. The paper rejects placing the split before the embedding: one-hot token vectors would make the permutation easy to recover.

TEE variant. To address a server that obtains a user’s capabilities, the paper adds an optional trusted execution environment that confines four simple operations to an enclave: the embedding, the first layer’s masking and softmax, the last layer’s normalization, and the user-side reverse transform. A one-time random scaling per query makes repeated prompts produce different transformed embeddings. The paper extends the scheme to Pre-LN models, RMSNorm, GeLU, SwiGLU, LLaVA and Mixtral.

Results reported in the paper

The model server used four NVIDIA A100 GPUs. User devices were an Intel i7 PC, an NVIDIA Orin module and a MacBook Pro, depending on the testbed.

Measurement	Result as reported
Equivalence	On 10,000 samples per model, six model series from 4M to 70B parameters gave identical top-1 predictions to the plaintext models. The remaining output differences are attributed to floating-point error.
Latency	31.7 ms on LLaMA2-7B, as stated in the abstract.
Throughput	With a batch of 100 and 100 tokens per sample: 45,366 tokens/s for GPT2-124M and 3,738 tokens/s for LLaMA2-7B.
TEE overhead	With the enclave holding only the first- and last-layer operations, latency increased by 2.6% to 9.4% across the evaluated models, and by 9.38% on a 70B model.
Production deployment	Deployed on the authors’ proprietary 70B model. In a three-month online test, STIP added 12% latency relative to the plaintext service.
Attacks	Brute force and known-plaintext analysis; distance-correlation tests on parameters and embeddings; model extraction by regression; a fine-tuning attack on the transformed model.

The proofs of Theorems 1 and 2 are in the appendix, which the paper marks as supporting material that has not been peer-reviewed. The authors describe their leakage bound as statistical and “weaker than absolute cryptographic secrecy”.

Limitations stated in the paper

The developer and the server must not collude.
All parties are assumed semi-honest.
A permutation recovered through a known-plaintext attack exposes that user’s later embeddings; the paper recommends rotating the permutations, in the extreme case per query.
The TEE variant inherits the trust assumptions of the enclave, including side channels.
Output privacy requires one device-cloud round trip per generated token.
The scheme does not apply to convolutional or recurrent layers.
The embedding table is exposed to users.

Relationship to kLLM

kLLM’s perm transform follows STIP’s feature-space parameter transformation and generalizes it to the structure of current models. It also departs from STIP at the input boundary.

kLLM component	STIP counterpart	Relationship
Residual-stream signed permutation under `perm`, shared by all layers	One residual permutation for all layers	Derived from STIP; kLLM adds random signs.
Query and key transform: permutation of whole grouped-query attention head groups with quarter turns of RoPE pairs	Per-layer permutation on a single-head formulation without RoPE	Same role; the construction for multi-head, GQA and RoPE models is kLLM’s.
MLP intermediate permutation, applied to `gate_proj` and `up_proj` alike	Per-layer feed-forward permutation	Derived from STIP.
Mixture-of-experts expert permutation, FP8 block-structured permutations	Router transform only; no FP8	kLLM’s.
Dense orthogonal transform under `orth`	None; STIP uses permutations only	Not derived from STIP.
Permutations derived from a 256-bit model key	Random generation by the model developer	kLLM’s.
Permuted vocabulary on the embedding and the LM head; the GPU receives permuted token ids	Embedding on the user’s device; the cloud receives a permuted embedding	A deliberate departure. It gives a stock-engine interface without a client-side embedding, at the cost STIP warns about.

The last row matters for security. STIP keeps the embedding on the user’s device because permuted one-hot tokens make the permutation easy to recover. kLLM sends permuted token ids to the engine so that SGLang and vLLM run unchanged. kLLM measures the consequence on Qwen3-8B: an operator who reads GPU memory faces a substitution cipher over tokens, and 23.8% of tokens are decoded by unigram frequency alone. STIP’s analysis of user-data leakage therefore does not transfer to kLLM. kLLM also uses one key per sealed model and does not rotate permutations. The recovery of the transforms and Π of a public model, or of a fine-tune of one, from the sealed weights is likewise a kLLM measurement; the STIP paper does not evaluate an attacker who holds the public checkpoint from which the transformed parameters derive. See Attacks and Claims.

The trust structure differs as well. In kLLM’s confidential-VM profile, the input and output transforms run inside the cloud’s confidential VM, and trust rests on attestation, not on non-collusion between a developer and a server. That profile relies on kLLM Enterprise components, has not yet been validated on TEE hardware, and does not protect GPU memory. In client-keyed mode, the developer and the data owner are the same party, and the claim is limited to logs and relayed payloads. See Deployment profiles and Client-keyed mode.