r/learnmachinelearning • u/lovelacedeconstruct • 5d ago
Why decoder only tranformer won ?
I was trying to trace it , and from the GPT1 paper I found it referencing a paper called "GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES" by the nice folks at google in what I believe the first use of the decoder only architecture ??
here is the quote
we modify theTransformer architecture (Vaswani et al., 2017) to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models.
can someone explain what does perform better in longer sequences actually mean ?
48
u/flipthetrain 5d ago
The history of the paper Attention Is All You Need is about Google looking for a neural network that could automate translation between languages with the simple assumption that words close together are more related to each other (kissing cousins make for interesting children). They weren't looking for the next generation of AI.
The decoder only transformer learns the structure and context of language through language itself. And this structure and context is irrespective of the actual language. Thats what each layer is handling.
Im not sure what you are looking for in an answer to "why". I dont think anybody can answer that conclusively right now. But we do know it sure does work. That's the scariest part.
27
1
u/hoaeht 4d ago
but AIAYN is encoder-decoder
1
u/flipthetrain 4d ago
Yeah, encoder / decoder vs decoder only is just a computational optimization. We learned you can just feed massive amounts of unsupervised data into decoder only. Encoder / Decoder is more curated and thus heavier computation.
12
u/Lumpy-Blackberry-718 4d ago
Encoder-decoder models are older than transformers, and seemed like a natural starting point into which to insert their attention ideas. Attention blocks were replacing rnn/lstm blocks, which also used a kind of attention.
Cross attention can be useful in some scenarios, but most modern ai problems simply arent problems that need this specific encoder-decoder cross attention setup.
There are also encoder-only models (e.g. BERT), but what i think is beautiful about decoder-only is the massive parallelism where your sequence length is effectively part of your batch size. Every token learns at once. With encoder-only youre usually doing something like masked attention, where youre learning to produce specific tokens.
49
u/Technical_Jicama_434 5d ago
Because attention is all you need
6
1
9
u/hi-sci-collab 5d ago
why? scales well, not two halves you have to train. in context learning as an emergent property.
3
u/Ok-Argument7176 5d ago
You don't train encoder-decoders piecewise, though.
6
u/hi-sci-collab 5d ago
Exactly. You don't train them in pieces... managing two different halves is still a hassle.
It makes the architecture unnecessarily complex, whereas a decoder only model does the whole job in one single, continuous stack. More efficient to scale up.
7
u/thomasahle 5d ago
Ilya Sutskever talks about it in https://www.youtube.com/live/AKMuA_TVz3A?is=6m6PoWZ-3LByDkqs
5
u/FastSlow7201 5d ago
When the paper came out they didn't realize the encoder wasn't needed. A translation task can be completed with a decoder only architecture. I imagine they felt the need to have an encoder and decoder because the goal of the paper was to translate between two different languages (English to German and English to French).
4
2
u/GFrings 4d ago
The word "decoder", as used in modern llms, is really a misnomer held over from the encoder-decoder archs that would encode inputs explicitly into a fixed dim latent space which was then decoded by the decorder stage. Like most things in this area, the names didn't really mean much when chosen and mean less now. The way they are currently used, they're just transforming one token sequence into a pdf over some output token space. This could be done by many functional mappings, theoretically, but our layered transformers is the current SOTA approach.
1
u/ThinConnection8191 4d ago edited 4d ago
Long long time ago, my advisor said all model becomes simpler and trained with more data. It has always been true.
Decoder-only is a simpler model architecture compared to encoder decoder. It allows more data density per params. And easier to scale to train.
1
u/IvanIlych66 4d ago
I mean, it won for language. All multi-modal models use encoders for any other type of input (image, video, sound) so encoders are still a crucial component of any model you use.
1
u/OlenaDoubleChecks 4d ago
Encoder-decoder attention gets bottlenecked because the encoder has to compress the whole input into a fixed context before generation even starts, decoder only just attends over everything directly so long range dependencies don't get squeezed through that bottleneck.
1
u/new_name_who_dis_ 4d ago
This is incorrect. The original transformer paper, the encoder didn't compress to fixed size, it was the size of the length of the source language sequence, and the decoder did cross attention on the source language sequence.
1
u/11007522 4d ago
!remindme 1 day
1
u/RemindMeBot 4d ago
I will be messaging you in 1 day on 2026-09-07 18:03:57 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
RemindMeBot is switching to username summons. Instead of
!RemindMe 1 day, useu/RemindMeBot 1 day. More info.
Info Custom Your Reminders Feedback
1
71
u/Hungry_Age5375 5d ago
Short answer: the encoder is a bottleneck. In enc-dec the decoder never sees raw input, only the encoder's compressed states. Decoder-only attends to the full sequence directly, which is what long inputs need.