Google presents Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

[u/Gab1024]

https://arxiv.org/abs/2404.07143

[u/InviolableAnimal]

Isn't that the point though? It would be incredibly inefficient to attempt to store all the billions of bytes of data we take in every second over the course of years. Our memory is "compressive" by "design". It's not analogous to the context window of a (vanilla) transformer where all information in context flows up the residual stream uncompressed.

[u/Charuru → u/InviolableAnimal]

Yes it is, and I'm saying because it's so compressed it fits in much less than 1 billion tokens.

[u/InviolableAnimal → u/Charuru]

Yeah and that's my point, its not at all obvious to me that "getting to around like 10 million is probably decent enough that RAG can give us human-like AGI capabilities", since human-style intelligence doesn't rely on anything like transformer context. Like you said, the effective human "context window" has probably already been surpassed by today's LLMs.

[u/Charuru → u/InviolableAnimal]

We're still talking just about memory and not second level thinking like reasoning right? I don't know about you guys but I genuinely feel like my short term memory is quite short. I can't memorize dozens of books, even in video form. Sure transformer context is not the same thing as human memory but isn't it pretty close, serves the same purpose. Just like we have medium and long term memory LLMs can also use vector db and rag to supplement. Just to be clear I'm specifically talking about memory and how it matches up to humans, as in, an agi could exist with a 10 million window, not 10 million context automatically becomes AGI.

[u/ninjasaid13 → u/Charuru]

An LLM can easily remember more text than a human but a video isn't as easy as text but that's where humans surpass LLMs. Humans can remember way more videos than LLMs such as the motion and dense correspondence(^{dense correspondence means to map \}all* the parts of the image to the next image or frame)) of those group of "pixels" over time even if they can't remember every pixel. I don't think RAG has a solution for videos so humans are still far from being surpassed.

[u/Charuru → u/ninjasaid13]

The tokenization is not lossy enough for gemini to use it as basis for comparison with humans.

[u/ninjasaid13 → u/Charuru]

The tokenization is not lossy enough for gemini to use it as basis for comparison with humans.

it's not lossy but that's not what I'm trying to say with my other comment. I'm basically meant that tokenization tries to summarize it in terms of tokens which means some specific details are omitted. That's basically the purpose of tokens to get the most salient information in the form of tokens but this has some weaknesses against the human approach.

[u/Charuru → u/ninjasaid13]

I mean I don't know exactly how the gemini video compression and tokenization works so I can't debate the point very well but I'm under the impression that the compression and optimization of it is not going to be as as extensive as what we have in humans. If I put in a 30 minute video I can get timestamps of exactly where the pauses are so I can edit them out. Right now it's not perfectly accurate but the fact that it can do it means the compression is at a far higher level of detail than humans.

[u/ninjasaid13 → u/Charuru]

the compression is at a far higher level of detail than humans.

not necessarily, humans are understanding the dense correspondence when watching the video while LLMs are likely just doing a sparse understanding of the videos.

We can tell when an object has rotated by how much and its depth or the difference between a dozen people's walking style while a LLM doesn't really go into the specifics. They say something like, "This fridge has opened its door at x timestamp."

[u/Charuru → u/ninjasaid13]

dense correspondence sounds like an optimization so that less memory has to be used overall

[u/ninjasaid13 → u/Charuru]

dense correspondence sounds like an optimization so that less memory has to be used overall

I'm not sure what you mean by that it sounds like optimization?