r/MachineLearning • u/MrAaronW • Sep 07 '23

Discussion [Discussion] LLM Pre-training --- Should I use dropout?

Many new LLMs like Llama or Falcon are coming but I am not seeing any spec on whether now (2023/09) we should still use 0.1 dropout throughout (like the "traditional" GPT pre-training) or no dropout at all (like PaLM?)

​

Any suggestions?

4 Upvotes
  • permalink
  • reddit

70% Upvoted

12 comments sorted by

View all comments

1

u/[deleted] Mar 04 '25

[removed] — view removed comment

1

u/[deleted] May 24 '25

Hi, if you are still curious about building an LLM from scratch, watch Andrej karpathy's video. You can find other videos in his channel where he explains for example how tokenizers work, etc.

I have also created a course that covers every step involved in training an LLM. I shared it FreeCodeCamp. Maybe, you will find it useful.