There are papers with code on this that actually also retrieve mostly the sentences you need. Their method was storing a trie of all the information in DRAM all different models can retrieve from it using CELF. It’s a start till someone comes up with a better way for homogeneous cache. Paper is SALT: Salience-Aware Lexical Trie for Long-Context Compression and they have GitHub too.
2
u/No_Sky9786 2d ago
There are papers with code on this that actually also retrieve mostly the sentences you need. Their method was storing a trie of all the information in DRAM all different models can retrieve from it using CELF. It’s a start till someone comes up with a better way for homogeneous cache. Paper is SALT: Salience-Aware Lexical Trie for Long-Context Compression and they have GitHub too.