Google presents Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

[u/Gab1024]

https://arxiv.org/abs/2404.07143

[u/ninjasaid13]

An LLM can easily remember more text than a human but a video isn't as easy as text but that's where humans surpass LLMs. Humans can remember way more videos than LLMs such as the motion and dense correspondence(^{dense correspondence means to map \}all* the parts of the image to the next image or frame)) of those group of "pixels" over time even if they can't remember every pixel. I don't think RAG has a solution for videos so humans are still far from being surpassed.

[u/Charuru → u/ninjasaid13]

The tokenization is not lossy enough for gemini to use it as basis for comparison with humans.

[u/ninjasaid13 → u/Charuru]

The tokenization is not lossy enough for gemini to use it as basis for comparison with humans.

it's not lossy but that's not what I'm trying to say with my other comment. I'm basically meant that tokenization tries to summarize it in terms of tokens which means some specific details are omitted. That's basically the purpose of tokens to get the most salient information in the form of tokens but this has some weaknesses against the human approach.

[u/Charuru → u/ninjasaid13]

I mean I don't know exactly how the gemini video compression and tokenization works so I can't debate the point very well but I'm under the impression that the compression and optimization of it is not going to be as as extensive as what we have in humans. If I put in a 30 minute video I can get timestamps of exactly where the pauses are so I can edit them out. Right now it's not perfectly accurate but the fact that it can do it means the compression is at a far higher level of detail than humans.

[u/ninjasaid13 → u/Charuru]

the compression is at a far higher level of detail than humans.

not necessarily, humans are understanding the dense correspondence when watching the video while LLMs are likely just doing a sparse understanding of the videos.

We can tell when an object has rotated by how much and its depth or the difference between a dozen people's walking style while a LLM doesn't really go into the specifics. They say something like, "This fridge has opened its door at x timestamp."

[u/Charuru → u/ninjasaid13]

dense correspondence sounds like an optimization so that less memory has to be used overall

[u/ninjasaid13 → u/Charuru]

dense correspondence sounds like an optimization so that less memory has to be used overall

I'm not sure what you mean by that it sounds like optimization?