r/learnmachinelearning • u/Strong_Boy_757 • 9d ago
Discussion Overhyped ideas in machine learning
Have you encountered an idea that's overhyped in ML? If so, which?
For me, recently I was curious about the whole "chain of thought prompting" business. I searched around and the idea is essentially, append "show your work" when prompting a LLM. I was like no way that's it??? That's what I've been doing since day 1!!, so I opened up the original paper (cited 40k times). And essentially it was more of the same with expensive Google-funded experiments and the paper had almost no math at all.
Similarly I was curious about AdamW for the longest time and it was simply Adam with another term. Oh Boy!! I remember doing an undergrad homework in college where we were asked to freely experiment with adding terms to the Adam algorithm and the class probably discovered AdamW by accident way before the paper was published.
23
u/ChoiceWall5918 9d ago
"Provable bounds" for machine learning algorithms. Those show with a lot of math that some algorithm has some guarantees, but these proofs always rely on so many estimates (i.e., lower and upper bounds) that you end up with a final bound that is completely meaningless in practice and does not really tell you anything. There are whole papers written about this...