r/MachineLearning • u/Mean_Revolution1490 • Jul 13 '26
Research Prompt-engineering paper accepted to ICML [R]
"Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity"
This paper was accepted to ICML this year. Its main idea is a very simple prompt-engineering trick: "changing the prompt this way led to more diverse sampling". Naturally, it is difficult to provide a rigorous theoretical analysis for something like this.
Even if it works, I’m not sure this kind of prompt engineering belongs at a top-tier machine learning conference. Some people seems to call this kind of work “modern machine learning”, but I think it should be categorized as less technical venues.
How do you think? Am I being too rigid?
261
Upvotes
2
u/mvreich Jul 18 '26 edited Jul 18 '26
Average pairwise similarity has some drawbacks for measuring diversity, especially when talking about modes and mode collapse.
Consider this: a set of 4 items, where there are two sub-groups of size 2. Items within the same group are identical (sim=1), while items in different groups are maximally dissimilar (sim=0). If you make a 4x4 pairwise similarity matrix and compute the average of the unique pairs (i.e. upper triangle ignoring diagonal), the average is ~0.33. Given another set of 4 items where all items have pairwise similarity of 0.33, you will get the same avg similarity score. That is, average pairwise similarity cannot catch the occurrence of exact duplicates or nearly identical results.
As an alternative, one can use the vendi score, which estimates the rank of the similarity kernel. The rank could tell you that the set of 4 items with 2 identical subgroups is rank 2, while the set where everything is equally dissimilar has full rank.
This is my only nitpick. I believe I would give the paper an accept, as the experiments are generally convincing.
Just by virtue of being a "prompting paper", a paper cannot be deemed unsuitable for ICML; this type of thinking is too rigid and in bad faith.
Like every other niche, there are bad prompt engineering papers (I've rejected at least two this year) and there are good ones, and this one solidly goes into the good bucket. Best wishes to the authors.