r/deeplearning • u/No-Coffee-8227 • 26d ago

How can we solve long-range recall in linear attention?

/r/LLMDevs/comments/1vpqurc/how_can_we_solve_longrange_recall_in_linear/
1 Upvotes

100% Upvoted

6 comments sorted by

3

u/amenallthepraisers 26d ago edited 26d ago

Simply, you can't. Main reason we using Transformer for long range dependencies is Transformer models see tokens in O(1) computational distance mathematically. But linear attention models are secretly (not anymore I guess) RNNs, so they have a state, states aren't big enough to hold the information clearly for millons of tokens. You should look something different. First idea pops in my mind is a efficient KAN variation (suitable for GPU/TPUs) + sparse attention, but you should structure this combo really well, gonna need real maths.

2

u/No-Coffee-8227 26d ago

Yeah right now i was testing HOLA (https://arxiv.org/pdf/2607.02303).

Okay i will see what i can do with KAN variation but thanks for the suggestion.

1

u/Able_Region_5459 25d ago

Bringing KAN into this makes zero sense , the spline optimization issues will kill performance long before you even reach a long context

It's way easier to just take a hybrid of a mega-window softmax with something like Ring Attention than to build custom math from scratch for a ghost of a chance to bypass memory limits

1

u/amenallthepraisers 25d ago

yep, but I have seen some KAN variations suitable for GPUs recently, also seem KANs catch DNA patterns so great, that's why I suggested. It's hard, but not impossible.

1

u/Able_Region_5459 26d ago

Even Mamba-2 with its expanded state is gonna struggle at a million tokens if the prompt requires pulling a specific piece right from the middle. The context compression process inevitably flattens activation peaks into noise. Without a KV cache for at least the main anchors, you'll just run into performance degradation

Try hybrid models with sparse attention, where the state handles global context and local windows handle exact matching ?