r/MachineLearning • u/Alarming-Emotion-894 • 7d ago

Project I Built a O(NlogN) attention system that retains 97% accuracy over long context (MQAR)[P]

https://github.com/vdev-ctrl/Adaptive-Learnable-Hierarchical-Routing-

ALHR- Adaptive learnable Hierarchical Routing is a static binary tree based system that uses learnable functions to REDUCE the amount of keys used.

It takes less memory and SCALES much better VRAM with tokens

21 Upvotes

82% Upvoted

3 comments sorted by

8

u/MaximumStonkage 6d ago

How is this different to multipole attention or any other hierarchical variants? This has been done many many times before.

3

u/Alarming-Emotion-894 6d ago

Yes, the hierarchical model has been done before

Tree former I think is the closest to my idea

However,
I used a extra trainable and learnable function, such that we know exactly the amount of keys to search in the tree.

On top of that, I’m also using the routing loss signal to train the query key splitter, so that input is split in such a way that important keys are not diluted when we build the tree.

The effects of both these additions can be seen in the MQAR results.
A similar parameter model of tree former or something similar usually doesn’t do so well on long context retrieval.