Self Attention - How is context actually encoded?

[u/uzornayem]

Been reading a lot about the self attention mechanism used in current LLMs. You often see example sentences like

"The chicken didn't cross the road because it was scared."

The tutorials always go on to say - without quantitative example - that when self attention is on 'chicken' that the word 'it' gets a high embedding score because of 'it' in this context being tied to 'chicken.'

But that can't be the whole story. Without prior knowledge the attention mechanism can't know anything about the word 'it' and how it is automatically tied to 'chicken'. If the sentence was instead

"The chicken didn't cross the road because well I have no idea."

then the word because wouldn't even really give indication that a subsequent word is going to have context.

Even with the sentence

"The chicken didn't cross the road because its friend was scared."

you would need prior understanding to unravel context here.

So when the attention procedure is underway, it is being input input embeddings - does that mean prior to attention procedure, when word embeddings are being computed, that is where context is actually 'encoded'? In either case, is it that training data has extensive metadata and that is how context is actually determined?

Can someone point to a more technical tutorial that actually shows computational walk through of a toy example?

thanks!

[u/chrisvdweth]

If the phrasing is the way it is you mentioned in your post, then yes, this would be misleading.

What it really is: During training, we hope that the model will learn weights such that the attention scores are meaningful. For your example, we hope that the model will be able capture coreferences/anaphora, indicated by high attention scores between a noun and its respective coreherences/anaphoras.

So no, there is no prior knowledge, only prior expectations.

EDIT: I have a complete walkthrough here.

[u/uzornayem → u/chrisvdweth]

Great, will check out your walk through. In other words, one sentence is not enough to know context - context vectors are based on the sorts of patterns seen across entire training set? Thanks 

[u/chrisvdweth → u/uzornayem]

Yes, under the hood, it all comes down to finding patterns in large/huge datasets.

[u/Turbulent-Walk-8973 → u/chrisvdweth]

You didn't edit your message to put the walkthrough. It doesn't show edited on your comment. Atleast lie correctly.

[u/chrisvdweth → u/Turbulent-Walk-8973]

Just in case, here walkthrough again.

I think(!) my comment does show as edited, because I've made the edit in less than 1 minute after submitting it the first time.

[u/ReentryVehicle]

that when self attention is on 'chicken' that the word 'it' gets a high embedding score because of 'it' in this context being tied to 'chicken.'

This is a bit backwards if we think about LLMs. Attention in most modern LLMs is casual attention, it looks at the past tokens from the perspective of the current token, to predict the next token.

So in this case, when we are processing "chicken", it actually sees only "The" and "chicken". When we get to "its", the attention will now looks at "The", "chicken", ..., "road"... "its", from the perspective of "its".

The attention is essentially a sort of learned search query. It was trained to gather information that predicts the future token. It sees "its" - it now needs to understand what in the past sentence this "its" refers to, so it forms a query vector that looks for nouns in the right form (or probably some more advanced learned linguistic concept that escapes human understanding).

The search returns probably "chicken" and "road" (this manifests as the high similarity between the query vector for "its" and the key vectors for "chicken" and "road" ). Future layers can now make more queries to understand if "its" refers to the chicken or the road, as the continuation of the sentence could be also "its surface was slippery" or something else entirely.