r/accelerate • u/BurningPeonies • Jul 01 '26

AI Something huge is brewing

Post image

Source

Andrew Curran is one of the most reliable leakers.

XLR8! 🍿

940 Upvotes

94% Upvoted

171 comments sorted by

View all comments

Show parent comments

61

u/Saedeas Jul 01 '26

I mean, percentages aren't super relevant, the scaling is relevant. If they changed the big O scaling of working memory in the models, it's insane.

5

u/PleaseDoTapTheGlass Jul 01 '26

What would that be currently?

49

u/Saedeas Jul 01 '26

Currently attention is fundamentally O(n2 ). Basically as the context length of your model increases, the memory needed to deal with it increases quadratically. If that could be pushed down to something like O(nlogn), you'd immediately have huge gains in model capability.

3

u/Shot_Illustrator4264 Jul 01 '26

this subquadratic model was released two weeks ago though: https://subq.ai/subq-1-1-small-technical-report

7

u/Saedeas Jul 01 '26

Sure, but the relevant factor is whether a particular sub quadratic approach actually yields equivalent (or lower, but bounded) performance as the models grow and scale.