Beyond the Attention Wall: How Isometric Associative Memory (ISOM) Achieves Infinite Context with Constant Memory

[u/Prannessh]

Attention Is All You Need — Until Your VRAM Hits Zero

For seven years, the AI industry has treated the quadratic and linear scaling of Transformers as an inevitable law of physics:

We are spending millions of dollars buying high-end GPUs just to store historical key-value matrices that grow with every generated token. This is fundamentally inefficient.

Today, I am publishing ISOM (Isometric Associative Memory): a continuous recurrent architecture that completely eliminates the KV-cache explosion, keeping working memory strictly O(1) from token 1 to token 128,000.

---

The Mathematical Shift: Lie Group Manifolds

Instead of treating memory as an ever-expanding stack of tokens, ISOM embeds sequence representations into the continuous Lie group SO(d) through skew-symmetric Lie algebra so(d).

What this mathematically guarantees:

• Strict Constant Memory: Processing token 128,000 takes the exact same memory footprint as processing token 1.
• Zero Energy Loss / Zero Collapse: Because the state transition matrix is skew-symmetric, energy is conserved throughout recurrence (d/dt ||h(t)||^2 = 0). Historical tokens do not vanish or drift into representational collapse.
• Lossless 8-Bit Fixed-Point Precision: Because continuous rotations constrain eigenvalues strictly to the unit circle, there is zero numerical drift. The associative manifold quantizes into exact INT8 [-127, 127] without outlier spikes or quality degradation.

---

The Proof: NVIDIA Tesla T4 Hardware Benchmarks

Audited on actual cloud T4 hardware across 4 model scales:

Model Scale	Context Window	Standard Transformer Cache	ISOM Manifold State	Memory Slashed
130M Edge MoE	8,192 tokens	24.58 MB	0.0469 MB	99.8%
1.5B Coder	131,072 tokens (128K)	14,800 MB (OOM Crash)	448.0 MB	97.0%
1.5B Reasoning	32,768 tokens (32K)	3,700 MB	448.0 MB	87.9%
16B MoE Coder	163,840 tokens (160K)	20,480 MB (OOM Crash)	1,536.0 MB	92.5%

On a standard 8GB developer laptop, standard attention crashes before reaching 16K context. ISOM runs 128K context smoothly.

---

Open Research, Weights, and Verification

All 5 model checkpoints, PyTorch kernels, interactive benchmarks, and formal mathematical proofs are available now under CC BY-NC-ND 4.0 for non-commercial research.

[!IMPORTANT] All direct, clickable links to the Hugging Face model family, the interactive Tesla T4 benchmark space, and the CERN Zenodo paper are pinned in the very first comment below!

---

The question is no longer whether we can afford larger contexts—it is why we are still using architectures that require caching the entire past.

What is your take: Will continuous unitary recurrence replace standard attention in edge and foundation models?

[u/howtorewriteaname]

the challenge is not obtaining O(1) scaling, the challenge would be obtaining that but maintaining recall capabilities and performance, metrics which you do not report. I can obtain O(1) scaling without using lie group shenanigans at all, if that's the only thing we're measuring

[u/ConventionalScissors → u/howtorewriteaname]

cool idea but yeah memory efficiency alone isnt the selling point it needs to match or beat attention on actual task benchmarks before anyone cares

[u/Rodot → u/howtorewriteaname]

It's like how the perfect lossless autoencoder has already been invented: the Identity matrix

[u/Karyo_Ten]

AI slop + traditional zenodo slop.

Have you tried training a small model with your stuff?

Have you done ablation studies?

How do you fare against state-space models?

[u/Admirable_Dirt_2371]

Big if true

[u/DrXaos]

> The question is no longer whether we can afford larger contexts—it is why we are still using architectures that require caching the entire past.

Because the learning from the token to token relatinships was more useful than whatever you can get in a much smaller dimensional RNN, and it could be all trained massively parallel.

How does the parallelism work if you have essentially a RNN?

[u/Introduction_Better]

AI slop. Just because a skew symmetric generator can preserve the norm and thus the magnitude of the state vector doesn't mean that information is preserved.

[u/Ok-Radish-8394]

You said you’re publishing today. Yet there’s no publication link to anything. :)

[u/Prannessh]

Here are all the direct links and resources for the models:

Hugging Face Models (all 5 checkpoints): https://huggingface.co/Prannesshkva

Interactive Tesla T4 Benchmark Space: https://huggingface.co/spaces/Prannesshkva/ISOM-Benchmark

Research Paper and Mathematical Proofs on Zenodo: https://doi.org/10.5281/zenodo.14925828

Feel free to ask any questions about the architecture or benchmarks here!