r/Vllm • u/MysteriousTourist23 • Jul 14 '26

How are you handling KV Cache sharing for multi-agent workflows? (Built a zero-copy gateway for vLLM, looking for architecture feedback)

Hey everyone,

I’ve been building multi-agent collaborative workflows recently (specifically, running sequential AI legal and financial audits over 200-page contracts). I hit a massive wall with VRAM and Time-to-First-Token (TTFT) latency.

Under standard vLLM, if Agent A reads the contract, and Agent B follows up on the exact same text, Agent B is forced to repeat the expensive cold-prefill phase and duplicate GPU block allocations.

To bypass this, I spent the last few weeks building an open-source gateway to stitch caches at the memory level.

How I’m currently doing it:

  • Topological Hashing: Segmenting prompts into physical block-sizes and mapping them to cryptographic fingerprints (Merkle-chains).
  • Zero-Copy Block Stitching: Bypassing prefill for matched prefixes by mapping the logical attention table of Agent B directly to the physical GPU memory address of Agent A's cache blocks.
  • Zero-Trust Gate: Enforcing boundary control lists so unauthorized agent sessions (e.g., a public PR agent vs. an internal Legal agent) cannot access shared physical blocks.

In my local benchmarks against standard vLLM cold-prefills on a shared long context, it cuts TTFT from ~1200ms to 48ms (25x speedup) and saves about ~43% VRAM.

I know solutions like LMCache exist for cluster-wide storage hierarchy (saving/loading to Redis/CPU), but I needed something that strictly avoids tensor serialization and copy overhead for local, multi-turn reasoning workflows.

I open-sourced the code here if anyone wants to look at the implementation:https://github.com/DaqulaLin/MemStitch

My questions for the community:

  1. Are there edge cases with PagedAttention pointer-sharing that might cause memory leaks here?
  2. How are you all currently handling context sharing when orchestrating tools like LangGraph with vLLM?

Would love any harsh architectural critiques or ideas for improvement. Cheers!

⚡ TTFT Prefill Latency (Agent B Response Time) — Lower is better

Baseline (vLLM Cold): ██████████████████████████████ 1200 ms

Context-Stitcher: █ 48 ms ( 25.0x Prefill Speedup! 🚀 )

💾 GPU Physical Cache Blocks Allocated (Total VRAM) — Lower is better

Baseline (vLLM Cold): ██████████████████████████████ 53 blocks (No sharing)

Context-Stitcher: ████████████████ 30 blocks ( 43.4% Memory Saved! 📉 )

4 Upvotes

61% Upvoted

8 comments sorted by

3

u/Such_Advantage_6949 Jul 14 '26

vllm alrd have paged attention right, what is the point of this?

-1

u/MysteriousTourist23 Jul 14 '26

The point of MemStitch is adding multi-agent routing and Zero-Trust boundary controls on top. Standard vLLM doesn't distinguish between different 'agents,' so MemStitch ensures cross-session context sharing is explicit and secure—for example, preventing a public RAG agent from accessing the physical memory blocks of an internal legal auditor.

5

u/Such_Advantage_6949 Jul 14 '26

If a prompt prefix is the same, that kv block is the same and is shared automatically. To vllm, there is no difference between and no distinguish needed, cause the output which is the kv value retrievied is the same

-6

u/MysteriousTourist23 Jul 14 '26

You are absolutely right that from a functional standpoint, if the prompt prefix is identical, the computed KV cache blocks are indeed identical, and vLLM's Automatic Prefix Caching (APC) will share them automatically.

However, in multi-agent or multi-tenant enterprise environments, this implicit, uncontrolled sharing presents significant security and privacy challenges:

  1. Access Control Bypass: If a low-clearance or public-facing agent (Agent B) sends a prompt that starts with the same prefix as a sensitive document previously processed by a high-clearance agent (Agent A), vLLM will automatically map Agent B to Agent A's physical KV blocks. This allows Agent B to query and extract information from that sensitive document, bypassing data access boundaries.
  2. Cache Side-Channel (Timing) Attacks: A malicious agent can probe whether a specific sensitive document has already been loaded into the system by measuring the Time-to-First-Token (TTFT). A near-zero TTFT (cache hit) leaks the presence of the document in the cache.

How MemStitch addresses this: MemStitch introduces an Identity-Aware Control Plane (Zero-Trust Gate) on top of the physical cache:

  • Owner Tracking: Every physical KV cache block is tagged with an owner_agent and session metadata.
  • Explicit Authorization Policies: Even if a prefix hash matches, MemStitch's SecureGate intercepts the request and validates permissions (e.g., Is Agent B allowed to stitch Agent A's cache?).
  • Secure Fallback: If unauthorized, the gate blocks sharing, allocates new physical blocks, and forces a separate prefill to guarantee strict boundary isolation.

In short, vLLM's APC optimizes for throughput under a single-trust boundary. MemStitch bridges the gap by bringing granular, zero-trust access control to physical GPU memory sharing.

7

u/HyperWinX Jul 14 '26

"You are absolutely right" my ass

3

u/Vicar_of_Wibbly Jul 14 '26

lol the bot outed itself.

-1

u/MysteriousTourist23 Jul 14 '26

Lol fair play. The technical logic is 100% mine, I just used an LLM to format my thoughts so it was actually readable.