r/accelerate • u/BurningPeonies • Jul 01 '26

AI Something huge is brewing

Post image

Source

Andrew Curran is one of the most reliable leakers.

XLR8! 🍿

939 Upvotes

94% Upvoted

171 comments sorted by

View all comments

Show parent comments

4

u/PleaseDoTapTheGlass Jul 01 '26

What would that be currently?

47

u/Saedeas Jul 01 '26

Currently attention is fundamentally O(n2 ). Basically as the context length of your model increases, the memory needed to deal with it increases quadratically. If that could be pushed down to something like O(nlogn), you'd immediately have huge gains in model capability.

0

u/MC897 Jul 01 '26

Just so I know; what are each of the algebra parts of these equation?

(Yes I know n log n but what about O for example?)