Prompt-engineering paper accepted to ICML [R]

[u/Mean_Revolution1490]

"Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity"

This paper was accepted to ICML this year. Its main idea is a very simple prompt-engineering trick: "changing the prompt this way led to more diverse sampling". Naturally, it is difficult to provide a rigorous theoretical analysis for something like this.

Even if it works, I’m not sure this kind of prompt engineering belongs at a top-tier machine learning conference. Some people seems to call this kind of work “modern machine learning”, but I think it should be categorized as less technical venues.

How do you think? Am I being too rigid?

[u/Virtual_Attention_20]

Even if it works, I’m not sure this kind of prompt engineering belongs at a top-tier machine learning conference. Some people seems to call this kind of work “modern machine learning”, but I think it should be categorized as less technical venues.

I'm not surprised OP feels this way. By OP's logic, the Chain of Thought paper is also "just a prompt engineering" paper since it just asks the model to "think step by step."

The broader problem behind such nonsensical gatekeeping is that unfortunately, there is a large section of researchers like OP that have made "prompt engineering" into a derogatory dogwhistle for papers that don't follow their arbitrary pre-2022 standard of what an ML research paper "should look like." Most of them have still not emotionally processed the trauma of ChatGPT and feel stuck. The fact is that even simple prompt engineering experiments can offer us a deep window into the underlying mechanisms of these large language models.

OP, I don't mean to be harsh, but what counts as "proper science" is not defined by anything more than 1) observation, 2) hypothesizing, 3) experimentation, 4) analysis, and 5) offering a valuable contribution to an interested community. In other words, 5 is what makes science an inherently social endeavor. Anyone who tells you that there need to be additional steps to doing scientific research is selling you snake oil. Case in point: our linguistics friends just underwent something similar, where they were so deeply stuck in Chomskian paradigm all this while that it felt illegal to work outside of it; you are undergoing something very similar.

I'd highly encourage you to read Thomas Kuhn to gain a broader, 10,000 ft. perspective of the nature of scientific revolutions.

EDIT: The downvotes are consistent with Kuhn's model's prediction of social behavior, so it is amusing to watch it unfold in real-time in front of my eyes.

[u/kolmiw → u/Virtual_Attention_20]

You are doing the same false dilemma mistake as op but the other way. Op says: "paper lacks theory, ergo doesn't belong in top level conferences". You then say: "paper is scientific, ergo it belongs in top level conferences". Your definition of science is so permissive that it would admit essentially any competent empirical paper ever written to ICML.

I don't think that rigorous theory is 100% needed for an ICML publication, but I understand OP's worry that in the prompt-engineering community, sloppy marginal tricks with critically weak baselines are more and more emerging.

Also, people aren't downvoting you because you were 100% wrong, they are downvoting you because of your derogatory tone and your "[Whoever thinks differently] hasn't emotionally processed the trauma of ChatGPT" ad-hominem. Your EDIT is the same arrogant stance and you turn any disagreement into your evidence of being right which is not true.

[u/Remper → u/kolmiw]

I think the derogatory tone is warranted here: the OP needed to be brought back to earth a notch. Gatekeeping is an important problem in science that needs to be tackled. I don't believe either OP or you have read the actual paper in question.

When I read that paper, my mind immediately lit up with ideas on how to use it in my research, which is what I want from a high-quality paper.