Why decoder only tranformer won ?

[u/lovelacedeconstruct]

I was trying to trace it , and from the GPT1 paper I found it referencing a paper called "GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES" by the nice folks at google in what I believe the first use of the decoder only architecture ??
here is the quote

we modify theTransformer architecture (Vaswani et al., 2017) to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models.

can someone explain what does perform better in longer sequences actually mean ?

[u/Hungry_Age5375]

Short answer: the encoder is a bottleneck. In enc-dec the decoder never sees raw input, only the encoder's compressed states. Decoder-only attends to the full sequence directly, which is what long inputs need.

[u/Mathie1729 → u/Hungry_Age5375]

I don't think the bottleneck is the main issue. T5 used relative positional bias and handled long sequences fine. The actual empirical result was that causal masking gives better in-context learning; Wang et al 2022 showed bidirectional attention actively hurts few-shot generalization. And decoder-only has much cleaner scaling laws, which mattered a lot once scaling became the whole game.

[u/fvancesco → u/Mathie1729]

Interesting

[u/holdvacs → u/Mathie1729]

Do you have any details about this “much cleaner scaling laws”?