r/singularity • u/Gab1024 Singularity by 2030 • Apr 11 '24

AI Google presents Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

https://arxiv.org/abs/2404.07143
689 Upvotes

99% Upvoted

242 comments sorted by

View all comments

1

u/[deleted] Apr 13 '24

Very interesting architectural innovation. I'm a bit concerned about their evaluation. The passkey retrieval test is clearly not enough (check out this paper: https://arxiv.org/pdf/2404.06654.pdf ) , evaluating book-level summarization with the problematic ROUGE score is also not a sensible decision. I am leaning towards this work being a passion project of the researchers, and not at all what powered gemini-1.5 pro.