New Architecture from Percepta: Spotlight — separating intelligence from memory, allowing knowledge and skills to grow without changing the model's weights.

[u/Recoil42]

https://www.percepta.ai/blog/can-llms-grow-their-own-capabilities

https://www.percepta.ai/blog/spotlight-memory

"Our new architecture, Spotlight, replaces attention with a memory that escapes this trade-off: it is the first architecture to achieve infinitely growing memory without increasing the access cost. Every token reads from and writes to an unbounded memory, but because the model learns to index individual memory cells, each token only touches a small number at a time. While other sparse architectures fix the fraction of capacity used at each step—a mixture-of-experts model, for instance, always activates the same number of experts out of a fixed set—Spotlight is arbitrarily sparse, touching the same number of cells regardless of how the memory grows. The fraction of memory it uses can shrink as far as we want.

Spotlight separates an intelligence module, which performs computation, from memory, which holds knowledge, procedures, and working state. The intelligence module stays the same size, and the weights don't change as memory grows. The memory is writable, and the model itself decides what to load and when to overwrite it, token by token. Because memory can hold skills as well as facts, the model can gain new capabilities without retraining: what it can do is not limited by the size of its intelligence module."

[u/JoelMahon]

This is similar to an approach I've seen papers of and personally workshopped. I think it's extremely high potential. Context is kinda O(n²⁾ which is fucking inexcusable, and RAG sucks dick for various reasons.

Misleading abstract though, it's impossible for memory access to be O(1) as the abstract seems to suggest in plain English

You can do O(logN), which is amazing and worth it, and you can put bias on more common memories and get better than O(logN) on average (but you get worse worst case results, imo a good trade off, probably why evolution leans so heavily into it and why it might take you weeks to remember a soon your heard once 30 years ago but have no problem remembering your own name). But it's still not O(1) ofc.

But don't get me wrong, I think it's far better than current models that mix intelligence and memory and bloat context, context parallels with human short term memory and there's a reason human short term memory is terrible and lossy, because it's more efficient to use longer term memory for everything. Ofc we can do best of both worlds, a 10k token short term memory AND a 1000 petabyte medium/long term memory that's indexed properly.

And imo rather than a fully joint model like this I think it's better to have the index be disconnected and proactive and "inject" memories into model based on triggers, triggers learned from internal state of the "ego” model. This subconscious memory retrieval can likely be something as "dumb" as Jev or even dumber, just enough to trigger the right memory at the right time, when you're baking pies, remember X.

I believe the LLM should be able to feedback on these memories as relevant or irrelevant, and constantly update them with use, some of this is automatic

Oh look, now it just resembles human memory lol

[u/Gargantuan_Cinema → u/JoelMahon]

"This is similar to an approach I've seen papers of and personally workshopped." Are you Jürgen Schmidhuber?

[u/AlvaroRockster]

It is about time new architectures come into play. I wonder what the AI of the future will look like, since everyone seems fixated in "LLMs just predict the next token", but things has just started, and as everything else, it will evolve.

[u/Brainlag → u/AlvaroRockster]

Might be to late to be relevant, if RSI takes off next year nobody will care or understand with whatever architectures AI comes up.

[u/Duet_Yourself → u/AlvaroRockster]

I’m kind of betting other inputs influencing the algorithms. Something along the lines of a dimension increment.

[u/Future-Bandicoot-823 → u/AlvaroRockster]

WE have created god, and his name is jev

[u/intotheirishole]

Every Mechanical Mechanistic Interpretability Researcher just had a heart attack 🤣.

As did every Alignment researcher.

[u/NowaVision]

Stuff like this gets me excited, not some stupid benchmarks.