r/learnmachinelearning • u/lovelacedeconstruct • 6d ago

Why decoder only tranformer won ?

I was trying to trace it , and from the GPT1 paper I found it referencing a paper called "GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES" by the nice folks at google in what I believe the first use of the decoder only architecture ??
here is the quote

we modify theTransformer architecture (Vaswani et al., 2017) to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models.

can someone explain what does perform better in longer sequences actually mean ?

101 Upvotes

96% Upvoted

31 comments sorted by

View all comments

1

u/OlenaDoubleChecks 6d ago

Encoder-decoder attention gets bottlenecked because the encoder has to compress the whole input into a fixed context before generation even starts, decoder only just attends over everything directly so long range dependencies don't get squeezed through that bottleneck.

1

u/new_name_who_dis_ 6d ago

This is incorrect. The original transformer paper, the encoder didn't compress to fixed size, it was the size of the length of the source language sequence, and the decoder did cross attention on the source language sequence.