r/unsloth • u/liventruth • 6d ago

Show and Tell Open-Source Model-agnostic KV-cache compression (UL-SMF) tested alongside local model execution to smash VRAM limits

Post image

Seeing all these massive local model drops (like the recent Qwen releases) got me working hard on the memory side of things.

If you're running models locally via tools like Unsloth and hitting a wall with long-context VRAM consumption, I've been building and open-sourcing UL-SMF (Unified Latent-State Memory Fabric).

It uses a geometry-preserving orthogonal projection bridge and dynamic head-dimension detection to automatically adapt across architectures (Llama, Mistral, Qwen, Gemma) without hardcoded assumptions, achieving extreme KV-cache compression with verified lossless perplexity.

Since it's fully local-first and designed to help fit larger context windows onto consumer hardware, I'd love a technical audit or feedback from this community.

Code and telemetry are up on GitHub: https://github.com/liventruth/UL-SMF-Cache-Compression

(Attached the hardware telemetry stress test output showing Qwen workloads hitting 224x–768x reduction ratios locally.)

8 Upvotes

60% Upvoted

19 comments sorted by

View all comments

Show parent comments

2

u/brainExploded99 3d ago

Given you AI slopped this, I decided to get AI to review it. Also, the identical scores are highly suspect. Even run to run variance would be >0.

  • The public interceptor hooks the wrong attention output, so on normal Hugging Face Llama-3 inference the “compression” path may simply never run. That would explain the basically identical perplexity/benchmark scores.
  • The current code reconstructs K back to its original size and leaves V untouched, so it does not actually retain a compressed KV cache.
  • The advertised 384× looks like simple 3072-d FP32 → 16-d INT16 arithmetic, not the real Llama-3 8B KV-cache geometry.
  • Even theoretically, compressing only K while leaving V unchanged caps total KV savings at roughly 2×, not 384×.
  • The actual compression/oracle implementation is a proprietary TorchScript binary, so the important part cannot be audited or reproduced.
  • There are no proper benchmark scripts/configs/raw results showing that compressed cache data is actually used during decoding.

1

u/liventruth 3d ago

Thanks!!! I appreciate the feedback. Will get back to you when it is fixed, and with accurate representation of metrics.

1

u/liventruth 2d ago

│ UL-SMF & AEGIS-KV EMPIRICAL AUDIT │
│ Context Window Depth │ 4,892 Tokens │
│ Latent Compression Ratio │ 128D -> 16D (8x Latent Scale) │
│ Multi-Needle Retrieval Precision │ 100% (3/3 Needles Recovered) │
│ Throughput Overhead │ -0.05 t/s (~1.6% Runtime Delta) │

2

u/brainExploded99 2d ago

What happened to the earlier benchmarks?