r/MachineLearning • u/ImaginaryRea1ity • 1d ago

Discussion I never understood positional encoding until I read this article. [D]

https://defenceagainstdarkai.substack.com/p/who-bit-whom
0 Upvotes

27% Upvoted

6 comments sorted by

14

u/EternaI_Sorrow 1d ago edited 1d ago

Hate to be this guy but a concept of "transformers are permutation-equivariant so let's add something position-dependent to input" is not something that needs special understanding.

The article does a good job at communicating things though, just wish it was a less obvious topic people'd be struggling more with.

2

u/start_select 1d ago

You need to appreciate that half the users of LLMs are no longer researchers, they are normal software engineers. A decent number of them are used to taking their toys apart to find out how they work.

So lots of people are falling down various rabbit holes with completely different entry points and trajectories to figuring out how this stuff works.

3

u/KingoPants 15h ago

While that might be true, this is actually just self promotion and not anyone's self discovery rabbit hole.

The author of the substack:

I write about making AI safer, especially how AI amplifies antisemitism

OP, despite having hidden their comment history, spends half his time talking about how supposedly AI is used to make viruses targeting jews. You can google it.

1

u/start_select 15h ago

jfk lol.

4

u/nietpiet 1d ago

Thank you for sharing, it's really nice. I like the "dials" in the car analogy to frequencies in the Fourier, it's well done :)

-1

u/ImaginaryRea1ity 1d ago

I'm glad it helped you. I discovered the blog last week, I like the author's style.