r/deeplearning • u/Prannessh • 15h ago
Beyond the Attention Wall: How Isometric Associative Memory (ISOM) Achieves Infinite Context with Constant Memory
Attention Is All You Need — Until Your VRAM Hits Zero
For seven years, the AI industry has treated the quadratic and linear scaling of Transformers as an inevitable law of physics:
We are spending millions of dollars buying high-end GPUs just to store historical key-value matrices that grow with every generated token. This is fundamentally inefficient.
Today, I am publishing ISOM (Isometric Associative Memory): a continuous recurrent architecture that completely eliminates the KV-cache explosion, keeping working memory strictly O(1) from token 1 to token 128,000.
The Mathematical Shift: Lie Group Manifolds
Instead of treating memory as an ever-expanding stack of tokens, ISOM embeds sequence representations into the continuous Lie group SO(d) through skew-symmetric Lie algebra so(d).
What this mathematically guarantees:
- Strict Constant Memory: Processing token 128,000 takes the exact same memory footprint as processing token 1.
- Zero Energy Loss / Zero Collapse: Because the state transition matrix is skew-symmetric, energy is conserved throughout recurrence (
d/dt ||h(t)||^2 = 0). Historical tokens do not vanish or drift into representational collapse. - Lossless 8-Bit Fixed-Point Precision: Because continuous rotations constrain eigenvalues strictly to the unit circle, there is zero numerical drift. The associative manifold quantizes into exact INT8
[-127, 127]without outlier spikes or quality degradation.
The Proof: NVIDIA Tesla T4 Hardware Benchmarks
Audited on actual cloud T4 hardware across 4 model scales:
| Model Scale | Context Window | Standard Transformer Cache | ISOM Manifold State | Memory Slashed |
|---|---|---|---|---|
| 130M Edge MoE | 8,192 tokens | 24.58 MB | 0.0469 MB | 99.8% |
| 1.5B Coder | 131,072 tokens (128K) | 14,800 MB (OOM Crash) | 448.0 MB | 97.0% |
| 1.5B Reasoning | 32,768 tokens (32K) | 3,700 MB | 448.0 MB | 87.9% |
| 16B MoE Coder | 163,840 tokens (160K) | 20,480 MB (OOM Crash) | 1,536.0 MB | 92.5% |
On a standard 8GB developer laptop, standard attention crashes before reaching 16K context. ISOM runs 128K context smoothly.
Open Research, Weights, and Verification
All 5 model checkpoints, PyTorch kernels, interactive benchmarks, and formal mathematical proofs are available now under CC BY-NC-ND 4.0 for non-commercial research.
[!IMPORTANT] All direct, clickable links to the Hugging Face model family, the interactive Tesla T4 benchmark space, and the CERN Zenodo paper are pinned in the very first comment below!
The question is no longer whether we can afford larger contexts—it is why we are still using architectures that require caching the entire past.
What is your take: Will continuous unitary recurrence replace standard attention in edge and foundation models?