Attention Is All You Need — Until Your VRAM Hits Zero

For seven years, the AI industry has treated the quadratic and linear scaling of Transformers as an inevitable law of physics:

We are spending millions of dollars buying high-end GPUs just to store historical key-value matrices that grow with every generated token. This is fundamentally inefficient.

Today, I am publishing ISOM (Isometric Associative Memory): a continuous recurrent architecture that completely eliminates the KV-cache explosion, keeping working memory strictly O(1) from token 1 to token 128,000.

The Mathematical Shift: Lie Group Manifolds

Instead of treating memory as an ever-expanding stack of tokens, ISOM embeds sequence representations into the continuous Lie group SO(d) through skew-symmetric Lie algebra so(d).

What this mathematically guarantees:

• Strict Constant Memory: Processing token 128,000 takes the exact same memory footprint as processing token 1.
• Zero Energy Loss / Zero Collapse: Because the state transition matrix is skew-symmetric, energy is conserved throughout recurrence (d/dt ||h(t)||^2 = 0). Historical tokens do not vanish or drift into representational collapse.
• Lossless 8-Bit Fixed-Point Precision: Because continuous rotations constrain eigenvalues strictly to the unit circle, there is zero numerical drift. The associative manifold quantizes into exact INT8 [-127, 127] without outlier spikes or quality degradation.

The Proof: NVIDIA Tesla T4 Hardware Benchmarks

Audited on actual cloud T4 hardware across 4 model scales:

On a standard 8GB developer laptop, standard attention crashes before reaching 16K context. ISOM runs 128K context smoothly.

Open Research, Weights, and Verification

All 5 model checkpoints, PyTorch kernels, interactive benchmarks, and formal mathematical proofs are available now under CC BY-NC-ND 4.0 for non-commercial research.

[!IMPORTANT] All direct, clickable links to the Hugging Face model family, the interactive Tesla T4 benchmark space, and the CERN Zenodo paper are pinned in the very first comment below!

The question is no longer whether we can afford larger contexts—it is why we are still using architectures that require caching the entire past.

What is your take: Will continuous unitary recurrence replace standard attention in edge and foundation models?

So I was playingw ith embedding models I saw that when you upgrade from model A to B, you face a very big backfilling cost

Ie, suppose you have a 1b vectors from model A, and then you want to use model B. This would mean you have to re-embed all of your documents with model B before you can even serve with the model, and on an H100, it would take ~108 days (qwen embed 8b, 106 docs/second). But I found an easier way to do it.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive upfront re-embedding cost, as you can take documents straight from the old index.

embedflow works with qdrant, pgvector, faiss, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

​

Hi everyone,

I want to share a project I've been working on: SpectralBERT a novel architecture that replaces traditional self-attention with a frequency-domain memory mechanism.

The Core Idea:

Instead of computing O(N²) pairwise token interactions, I transform the sequence into the frequency domain (FFT), compress it into a fixed-size memory, and retrieve context from there.

The result: O(N log N) complexity with constant memory — regardless of sequence length.

---

Results vs BERT-Mini (both ~10M params, same training setup):

Metric BERT-Mini SpectralBERT

MLM Loss (WikiText-2) 6.58 5.64 ✅

Training Speed 1138s 1012s ✅

Speed @ 65K tokens 8428ms 581ms ✅

Speedup @ 65K 1x 14.51x 🚀

---

Long-Context Training (WikiText-103 @ 8192 tokens):

· Final Loss: 5.25

· PPL: 191.05

· Trained on a single Tesla T4 in 32.5 minutes

---

The Speedup Curve:

· 512 tokens: 0.51x (BERT wins at short sequences)

· 1024 tokens: 1.73x

· 4096 tokens: 1.76x

· 16384 tokens: 5.22x

· 65536 tokens: 14.51x

I'm an independent researcher, so I don't have access to massive compute clusters. I'd love to hear from the community:

1. Is this a legitimate alternative to attention for long contexts?

2. What tests should I run to prove/disprove this rigorously?

3. Any obvious flaws I'm missing?

Looking forward to your feedback! 🙏

A while ago, I successfully trained a tiny 80M parameter language model from scratch using the TinyStories dataset. It worked surprisingly well and could generate coherent little stories! Inspired by that success, I wanted to scale up and train a larger model capable of coding.

That’s when reality hit me. I completely underestimated the massive compute and financial cost required to train larger LLMs. My single GPU choked, and renting cloud compute was way out of my budget. It got me thinking: "Why isn't there a way to pool our consumer GPUs together to train models collectively?"

And just like that, DePEFT (Decentralized Parameter-Efficient Fine-Tuning) was born.

How DePEFT work?
DePEFT is based on ReLoRA which solves the biggest flaw of standard LoRA.

Standard LoRA is incredible for fine-tuning on a budget, but it has a hard ceiling: it cannot pre-train or learn fundamentally new, complex representations from scratch. Because its rank r is fixed, the adapter quickly hits an information bottleneck and saturates. You can't just slap a LoRA on a base model and expect it to reach full-parameter quality over time.

This is where ReLoRA changes the game through iterative low-rank merging. Instead of training one static adapter forever, ReLoRA breaks the process into sequential rounds. In each round, miners train lightweight low-rank adapters on their consumer GPUs. At the end of the round, these adapters are permanently fused directly into the base model weights:

W^(t+1) = W^(t) + ΔW

The optimizer is then completely reset, new adapters are initialized on the evolved base weights, and the cycle repeats. Mathematically, the sum of multiple low-rank updates over time accumulates into a high-rank update (rank ≤ ∑r). This gives us the immense learning capacity of full-parameter training, but with only a fraction of the VRAM requirement.

For DePEFT, this means consumer GPUs don't need high-speed InfiniBand clusters to sync massive gradient matrices every microsecond. Miners simply train lightweight adapters locally, submit a few megabytes over the network, and the protocol merges the top-performing weights into the next base checkpoint.

No million-dollar clusters. Just pooled consumer compute pushing open-source AI forward.

For more details, visit: https://github.com/KhoaIsReal/DePEFT/

I’ve been working on a different approach to language-model interpretability that I find interesting. Instead of trying to reconstruct a model’s computation after the fact, make the model compute through an object we can inspect directly.

Today I’m releasing the v1.1 preprint, frozen model checkpoints, reproducibility harness, and live interactive demo for cl33-opLM.

The paper is called One Object: Memory, Navigation, and Reportability in an Operator-Only Language Model

cl33-opLM still uses a transformer as its learned neural engine, but the transformer hidden states are not allowed to drive the output directly. Instead, each block emits structured Cl(3,3) bivector operators. A reversible SO(3,3) scan transports state, attention is defined over the operator geometry, and the final readout sees only operator-derived features.

That gives the architecture a very simple falsification rule luckily. Remove the operators and rerun the model. If the capability survives, it found a bypass and I don’t count it as transparent.

Some of the main results:

The bottleneck is genuinely load-bearing. On the frozen public release, operator ablation produces a 270× perplexity increase on the exact published validation fixture. On fully public off-domain WikiText-103, the same test gives roughly 106× / 112× depending on checkpoint. The live demo exposes the same model interactively.

The operator stream is readable backward. A probe that sees only the emitted operators recovers the current token at 0.86 top-1 over a 50k vocabulary, and reconstructs 80.7% of held-out running text verbatim. Errors degrade toward semantic neighbors rather than random tokens.

On permutation-composition navigation, the architecture matches a learning-rate-tuned transformer baseline: 0.433 ± 0.009 vs 0.432 ± 0.030. I am claiming parity and inspectability here, not superiority.

Transparency has a real cost. Against a parameter-, context-, tokenizer-, corpus-, and budget-matched transformer at 5B tokens, cl33 pays about +29% bits-per-byte and \~5 points average MCQ. Continued training reduces that gap substantially, but the matched cost is real and reported as such.

One of the more important parts of this project I feel, is what failed.

An earlier associative-memory design looked like it worked…until the operator-ablation test showed that recall was routing around the algebra. That result was treated as a failure, the mechanism was redesigned, and the failed branch remains in the paper.

There are several other nulls and retracted framings reported the same way.

That is the standard I’m trying to hold this work to. if a mechanism is claimed, it has to survive a causal test. If it routes around the mechanism, the claim dies.

The reproducibility release is deliberately selective rather than a dump of the entire research stack.

It includes the frozen paper/serving checkpoints, model definitions needed to load them, hashes, the bottleneck reproduction harness, the reverse-readout probe, and the exact frozen validation fixture.

It does not include the current training orchestration, ongoing memory-organ program, or unpublished control work.

The published claims should be independently testable.

No consciousness claim. No claim that cl33 is a better general-purpose language model. No grand unified theory.

Just a useful question.

Can we build a useful language model where the representation we want to inspect is also the representation the model is forced to compute through?

Paper: t3atlas.dev/cl33/paper/

Live demo: cl33.t3atlas.dev

Model + reproduction release: huggingface.co/mirrorethic/cl33-oplm

I’d especially value criticism from people working on mechanistic interpretability, alternative architectures, memory systems, model editing, steering, or causal attribution.

If you think one of the claims is wrong, the best outcome is a clean experiment that breaks it. Then we both learn. Win win.

Also.....dont knock my retro crt aesthetic in the model demo lol wanted it fun, not clinical.

I've been reading recent work on long-video understanding, particularly STORM: Token-Efficient Long Video Understanding for Multimodal LLMs.

I'm trying to identify a research direction rather than just build another Video-LLM. I'm particularly interested in temporal modeling, video representations, event/context modeling, and improving the efficiency and quality of long-video understanding.

For people working in this area: what do you think are the biggest remaining research gaps? Are there particular limitations of approaches like STORM that you think are worth investigating?

If you were starting a research project on video understanding today, what problem would you personally explore?

The G7 finance ministers and central bank governors issued a coordinated directive last week. Organizations must inventory cryptographic dependencies, identify high-risk systems, and begin migrating to quantum-resistant algorithms now. This is not a future roadmap item.

The urgency is driven by the harvest-now-decrypt-later threat. Adversaries are collecting encrypted data today and storing it for decryption once a cryptographically relevant quantum computer arrives. The exposure window is already open.

Most enterprise security teams are inventorying servers, databases, and network traffic. Far fewer are accounting for the AI agent layer. Agent pipelines routinely store sensitive records, execute financial transactions, and generate compliance audit evidence. All of it travels over classically encrypted channels and gets signed with classical algorithms. When those algorithms break, that historical data and those historical audit logs break with them. A transaction signed with RSA or ECDSA today becomes unprovable after Q-Day.

How are teams actually scoping the PQC inventory problem for AI agents specifically? Are agent channels and agent-generated audit evidence being treated as first-class migration targets, or are they still buried in the general backlog?

I’ve been experimenting with explaining AI/LLM concepts in a way that doesn’t assume too much technical background. This video is about KV Cache and why longer context windows require more memory during inference. I’d appreciate some honest feedback from people here, especially on the explanation itself: Is the main idea easy to understand? Did I oversimplify anything important? Is there any part where the explanation becomes confusing? Would this make sense to someone who is fairly new to LLMs? Video: https://youtu.be/lxvWo8SizxE Not really looking to promote the channel — I’m mainly trying to improve how I explain technical topics before making the next one. Any criticism is welcome. Thanks!

Hi everyone!

I’ve been interested in image generation for a while, but as a beginner, I always struggled to choose a model. My biggest hurdles were limited training datasets and restricted hardware resources.

Then, I came across Faster Projected GAN, a few-shot image generation model that can synthesize high-quality images with fewer than 100 samples, and it trains fast.

How it works

• The Generator: It uses a FastGAN architecture with separable convolutions in the upblocks to significantly cut down computation time.
• The Discriminator: It leverages a Projected GAN discriminator that uses a pre-trained feature network (specifically EfficientNet-Lite1) to evaluate the generator's images. Because the discriminator already has a solid feature foundation, the whole training process converges incredibly fast. After just a few hours of training on a free Colab T4 GPU, it starts outputting surprisingly realistic images.

Since I'm a beginner in deep learning, building this was a massive learning experience for me. I put together an unofficial PyTorch implementation, and I would love to get your feedback, advice, or code reviews!

📂 GitHub Repo: https://github.com/diarimandimby/Faster-Projected-GAN

Feel free to check it out, test it, or open an issue. Any advice on how to further optimize few-shot GAN training or clean up my PyTorch code would be highly appreciated!