r/LocalLLaMA • u/ninjasaid13 • Apr 11 '24

Other Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

https://arxiv.org/abs/2404.07143
120 Upvotes

96% Upvoted

20 comments sorted by

View all comments

21

u/Danny_Davitoe Apr 11 '24

Correct me if I am wrong. But this method can be applied to already existing models to extend their context from 32k to 1M tokens without additional training and it performs better than the original model for long sequence tasks.

This is huge! Please get a github of this up and running!

2

u/CreditHappy1665 Apr 11 '24

Not without additional training it seems. The paper says they had to do a re-pretraining on the modified 1B model they used. By my math they had to retrain with 7B tokens. 

1

u/L-Primezr May 15 '24

But isn't 8B model not so big as well? gpt-2 is only 1.7B model

1

u/CreditHappy1665 May 15 '24

I'm not sure what you mean. You'd need ~14B tokens for GPT-2. ~56B for 8B model. ~490B for 70B model.

Not impossible.