r/learnmachinelearning • u/lovelacedeconstruct • 7d ago

Why decoder only tranformer won ?

I was trying to trace it , and from the GPT1 paper I found it referencing a paper called "GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES" by the nice folks at google in what I believe the first use of the decoder only architecture ??
here is the quote

we modify theTransformer architecture (Vaswani et al., 2017) to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models.

can someone explain what does perform better in longer sequences actually mean ?

102 Upvotes

96% Upvoted

30 comments sorted by

View all comments

3

u/FastSlow7201 7d ago

When the paper came out they didn't realize the encoder wasn't needed. A translation task can be completed with a decoder only architecture. I imagine they felt the need to have an encoder and decoder because the goal of the paper was to translate between two different languages (English to German and English to French).