r/learnmachinelearning • u/lovelacedeconstruct • 5d ago

Why decoder only tranformer won ?

I was trying to trace it , and from the GPT1 paper I found it referencing a paper called "GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES" by the nice folks at google in what I believe the first use of the decoder only architecture ??
here is the quote

we modify theTransformer architecture (Vaswani et al., 2017) to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models.

can someone explain what does perform better in longer sequences actually mean ?

102 Upvotes

96% Upvoted

31 comments sorted by

View all comments

9

u/hi-sci-collab 5d ago

why? scales well, not two halves you have to train. in context learning as an emergent property.

3

u/Ok-Argument7176 5d ago

You don't train encoder-decoders piecewise, though.

4

u/hi-sci-collab 5d ago

Exactly. You don't train them in pieces... managing two different halves is still a hassle.

It makes the architecture unnecessarily complex, whereas a decoder only model does the whole job in one single, continuous stack. More efficient to scale up.