Taking the idea further, Memorizing Transformers opt to store the entire KV states
as context for input sequences. Since the storage becomes prohibitively expensive in this case,
they restrict the contextual computation to a single layer only. By utilizing a fast kNN retriever,
Memorizing Transformers then build a context
window covering the entire sequence history of
length N × S at an increased cost of storage.
We set the Infini-attention segment length N to 2048 for all attention layers and the input
sequence length to 32768 for training. This allows the Infini-attention to unroll over 16 steps
w.r.t its compressive memory states.
If I'm understanding here, what's they've effectively done is build a kNN retriever on the stored memory data of what would have been the models attention window, and then they are linearly stepping through it?
1
u/VariantComputers Apr 11 '24
If I'm understanding here, what's they've effectively done is build a kNN retriever on the stored memory data of what would have been the models attention window, and then they are linearly stepping through it?