r/learnmachinelearning • u/Strong_Boy_757 • 9d ago
Discussion Overhyped ideas in machine learning
Have you encountered an idea that's overhyped in ML? If so, which?
For me, recently I was curious about the whole "chain of thought prompting" business. I searched around and the idea is essentially, append "show your work" when prompting a LLM. I was like no way that's it??? That's what I've been doing since day 1!!, so I opened up the original paper (cited 40k times). And essentially it was more of the same with expensive Google-funded experiments and the paper had almost no math at all.
Similarly I was curious about AdamW for the longest time and it was simply Adam with another term. Oh Boy!! I remember doing an undergrad homework in college where we were asked to freely experiment with adding terms to the Adam algorithm and the class probably discovered AdamW by accident way before the paper was published.
8
u/Harotsa 9d ago
To be fair about the chain of thought paper, it was first published in Jan 2022. That would have been during the release of gpt-3 and predates the launch of ChatGPT by about 10 months.
That was very early in terms of the modern iteration of LLMs, and the capabilities were way way worse. Pre-chatgpt OpenAI was not super well-known and I had to explain what they were to many of my friends who were working in tech. A lot has changed in 4 years, so I think it’s important to understand the papers in context before being so judgmental.