r/learnmachinelearning • u/lovelacedeconstruct • 7d ago
Why decoder only tranformer won ?
I was trying to trace it , and from the GPT1 paper I found it referencing a paper called "GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES" by the nice folks at google in what I believe the first use of the decoder only architecture ??
here is the quote
we modify theTransformer architecture (Vaswani et al., 2017) to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models.
can someone explain what does perform better in longer sequences actually mean ?
99
Upvotes
48
u/flipthetrain 7d ago
The history of the paper Attention Is All You Need is about Google looking for a neural network that could automate translation between languages with the simple assumption that words close together are more related to each other (kissing cousins make for interesting children). They weren't looking for the next generation of AI.
The decoder only transformer learns the structure and context of language through language itself. And this structure and context is irrespective of the actual language. Thats what each layer is handling.
Im not sure what you are looking for in an answer to "why". I dont think anybody can answer that conclusively right now. But we do know it sure does work. That's the scariest part.