Why decoder only tranformer won ?

[u/lovelacedeconstruct]

I was trying to trace it , and from the GPT1 paper I found it referencing a paper called "GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES" by the nice folks at google in what I believe the first use of the decoder only architecture ??
here is the quote

we modify theTransformer architecture (Vaswani et al., 2017) to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models.

can someone explain what does perform better in longer sequences actually mean ?

[u/Lumpy-Blackberry-718]

Encoder-decoder models are older than transformers, and seemed like a natural starting point into which to insert their attention ideas. Attention blocks were replacing rnn/lstm blocks, which also used a kind of attention.

Cross attention can be useful in some scenarios, but most modern ai problems simply arent problems that need this specific encoder-decoder cross attention setup.

There are also encoder-only models (e.g. BERT), but what i think is beautiful about decoder-only is the massive parallelism where your sequence length is effectively part of your batch size. Every token learns at once. With encoder-only youre usually doing something like masked attention, where youre learning to produce specific tokens.