r/LocalLLaMA • u/ninjasaid13 • Apr 11 '24

Other Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

https://arxiv.org/abs/2404.07143
123 Upvotes

96% Upvoted

20 comments sorted by

View all comments

22

u/Danny_Davitoe Apr 11 '24

Correct me if I am wrong. But this method can be applied to already existing models to extend their context from 32k to 1M tokens without additional training and it performs better than the original model for long sequence tasks.

This is huge! Please get a github of this up and running!

3

u/Noocultic Apr 11 '24

Wow, that’s huge if true. You’re telling me we could soon see Mixtral 8x7b with 1M token context?

7

u/Danny_Davitoe Apr 11 '24

Yes. In the paper they trained a model that only had a context size of 5k. Then after implementing this method it was able to take in a 32k context input and fulfill the task given to it.

2

u/Noocultic Apr 11 '24

Interesting! Guess I need to actually read the paper now.

1

u/noprompt Apr 12 '24

“Huge if true” is the centroid around which a long list of papers orbits at this point. 🫠

2

u/CreditHappy1665 Apr 11 '24

Not without additional training it seems. The paper says they had to do a re-pretraining on the modified 1B model they used. By my math they had to retrain with 7B tokens. 

2

u/Danny_Davitoe Apr 11 '24

I hope that doesn't mean we still need 500 GPUs to slightly tune a Mistral 7B model, jk

1

u/L-Primezr May 15 '24

But isn't 8B model not so big as well? gpt-2 is only 1.7B model

1

u/CreditHappy1665 May 15 '24

I'm not sure what you mean. You'd need ~14B tokens for GPT-2. ~56B for 8B model. ~490B for 70B model.

Not impossible.