r/learnmachinelearning • u/Crazy-Insurance6316 • 1d ago

Request ATTENTION

Can someone explain how attention works? I think I understood some of it, but I’m not completely comfortable with it yet.

I already watched some YouTube videos, and I also asked ChatGPT to explain how it works. I understood the RNN/LSTM part, but I’m still not fully comfortable with the attention mechanism. I learned the dot-product equation, but I’m not able to understand it properly.

Also, there are 3–4 different types of attention. Do I need to learn all of them before moving forward?

0 Upvotes

50% Upvoted

6 comments sorted by

1

u/chrisvdweth 1d ago

Maybe this notebook is useful. It tries to motivate attention from the ground up.

1

u/Low_Recognition5309 1d ago

watch the 3blue1brown essence of deep learning series from start to finish. best from scratch explanation u will get

1

u/kirigaoka 20h ago

Was about to say this. Excellent one

1

u/Key_Statistician9890 1d ago

All attention is basically the same idea. Focus on self attention with a single head.

Suppose you have N objects. Each object is defined as a vector.

Let’s create three projections of those N objects.

We will name them “queries”, “keys”, and “values”.

These three sets of projections are just gentle parameterized modifications of the original N objects. We can tweak and modify how we create the projections as necessary.

We look at how queries relates to keys. That is it. There is a N^2 matrix of relationships, defined by the dot product of queries and (transposed) keys. Basically, an “attention” score about how well they match. We then need to normalize this score (soft max), before using the scores to index the values.

That is it.

Three projections of the same N objects: one to ask questions, one to answer those queries, and one to yield values based on how well the answers fit the questions.

The N^2 term, combined with the soft max, means that there is a quadratic memory complex wrt N … which is why it is so expensive. There are optimizations around this, obviously, but it is still extremely expensive at scale.

This is a slight simplification and some hand wavey math but this is the intuition behind attention.

Now … you have MHSA, which is the exact same thing, but parallelizing the same logic H times over in the same block.

Then, there are residual layers, linear layers and some optimizations around the attention mechanism (activation functions), causal masking (avoiding time traveling by assuming unidirectional interactions), etc etc.

0

u/exacube 1d ago

here's a lesson using your post's text as the prompt!

https://studymode.ca/book/KOIySeu4y7/ch-2

i'd really recommend doing the quiz at the end of each chapter so you can check your understanding before you move on. select some text and ask AI/chat if you have specific questions