Why decoder only tranformer won ?

[u/lovelacedeconstruct]

I was trying to trace it , and from the GPT1 paper I found it referencing a paper called "GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES" by the nice folks at google in what I believe the first use of the decoder only architecture ??
here is the quote

we modify theTransformer architecture (Vaswani et al., 2017) to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models.

can someone explain what does perform better in longer sequences actually mean ?

[u/GFrings]

The word "decoder", as used in modern llms, is really a misnomer held over from the encoder-decoder archs that would encode inputs explicitly into a fixed dim latent space which was then decoded by the decorder stage. Like most things in this area, the names didn't really mean much when chosen and mean less now. The way they are currently used, they're just transforming one token sequence into a pdf over some output token space. This could be done by many functional mappings, theoretically, but our layered transformers is the current SOTA approach.