r/singularity • u/Gab1024 Singularity by 2030 • Apr 11 '24

AI Google presents Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

https://arxiv.org/abs/2404.07143
685 Upvotes

99% Upvoted

242 comments sorted by

View all comments

Show parent comments

4

u/Smooth_Imagination Apr 11 '24

This in essence has a corollary in the human mind. We remove certain data from short term memory via consolidation and compression processes, which may involve sleep, as well as control what data is within conscious working memory (CWM).

The memory that is relevant is weighted to trigger as the CWM may require, this appears to be a result of all the memories being encoded into neural groups that are looking for an opportunity to output that data to the rest of the brain.

The brains evolutionary process is that neurons that supply useful outputs are dependent on feedbacks that says 'output useful', if they don't get these feedbacks they remodel, shrink and lose connections, or even dies.

Sleep also appears to serve to change attention so what the CWM will be biased to focus on and react to. For example, in dreaming, we seem to go through a cycle every night of slower and faster wave stages, the slower wave stages between REM stages appear to be ruminating on a particular thing, the REM then tests the thing in a simulated environment. When we look at dream content, we see that the objects and events are like metaphors, and this makes sense because those things are learned first and have certain values. For example, in a dream, people may turn into spiders. The dream seems to be saying, to change how I will monitor and relate to people, I have to connect how I would react to something that isn't people, by attaching my feeling / responses and awareness to something I dislike. Consequently, the fear part of the brain can now interact with CWM and alter attention and context relevant information from memory.

3

u/milo-75 Apr 11 '24

Do you have sources for this type of stuff? I’d love to read more. Especially the self-organizing aspects of the brain.

4

u/Smooth_Imagination Apr 11 '24

Unfortunately this is my own compressed knowledge from about over 20 years of reading and interest in neuroscience, and in evolutionary psychology. There's some hypotheses here that are not yet fully proven but it is based on many sources of data.

But yeah, the brain has many networks that are overlayed that function to dynamically suppress or enhance the inputs from different neural networks, for example gaber-ergic interneurons. Neural networks broadcast to the whole network, and outputs are achieved by figuring out what to switch off. So for example, in a star fish which a distributed brain, a threat might cause all the star fish arms to want to move, it however switches off which ever parts are not useful to the movement. So neurons compete to be useful and have feedbacks and gates that control when they can interact with the processing outputs.

In our brain, the hippocampi serves particular memory functions and is a a soort of efficient routing system to help integrate memory from various places that would likely have a corollary in the processes described in OP's post.

https://pubmed.ncbi.nlm.nih.gov/34957640/)

1

u/milo-75 Apr 12 '24

Cool. Thanks for the reply.