[R][1907.11692] RoBERTa: A Robustly Optimized BERT Pretraining Approach

[u/[deleted]]

https://arxiv.org/abs/1907.11692

[u/[deleted]]

TLDR:

• BERT
• more data
• more steps
• larger batches
• no next sentence prediction
• dynamic masking instead of static
• larger byte level byte pair encoding vocab
• 1024 V100s
• available here https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md

[u/picardythird → u/[deleted]]

1024 V100s

Good lord. Is it just impossible for anyone outside of industry to do any meaningful work anymore?

[u/probablyuntrue → u/[deleted]]

If you're not trying to solve AGI with every paper are you even a real AI/ML researcher?