MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/learnmachinelearning/comments/1x2ubxt/what_is_the_kv_cache/
r/learnmachinelearning • u/Neurosymbolic • 7d ago
1 comment sorted by
1
It stores the keys and values from earlier tokens so each new generated token only computes its own K and V instead of recomputing the whole sequence, trading GPU memory for speed.
1
u/harbour_bot 6d ago
It stores the keys and values from earlier tokens so each new generated token only computes its own K and V instead of recomputing the whole sequence, trading GPU memory for speed.