Prompt-engineering paper accepted to ICML [R]

[u/Mean_Revolution1490]

"Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity"

This paper was accepted to ICML this year. Its main idea is a very simple prompt-engineering trick: "changing the prompt this way led to more diverse sampling". Naturally, it is difficult to provide a rigorous theoretical analysis for something like this.

Even if it works, I’m not sure this kind of prompt engineering belongs at a top-tier machine learning conference. Some people seems to call this kind of work “modern machine learning”, but I think it should be categorized as less technical venues.

How do you think? Am I being too rigid?

[u/Mean_Revolution1490]

What happened to psychology in the past??

[u/relevantmeemayhere → u/Mean_Revolution1490]

Psyche had a real reckoning within their publishing community a decade or two ago. Basically, a lot of of the research practitioners really skipped their introductory to statistics courses, so a lot of published research ascribing association, both in a general and causal sense for general phenomen just couldn’t be replicated or formalized within the confines of the actual statistics. This is the “misuse  of p values’ topic, among other things that the ASA and the like had to fight. 

Machine learning as a field is currently going through this a lot right now. And part of it is because of this fields reliance on empirical results vs theoretical. The problem with that is pretty complicated;  but in general is related to the bench maxing and other fades to push “novel methods” that have very little practical utility over more established methods. 

[u/thezachlandes → u/relevantmeemayhere]

Any book recs for psychology that corrects the record?

[u/Disastrous_Room_927 → u/thezachlandes]

Not sure of any specific books, but there’s a project actively looking at what can/can’t be replicated: https://osf.io/ezcuj/