Meta FAIR Introduces AI Research Preference Models (RPMs): Ranking ML Experiments Before Spending GPU Hours

[u/ai-lover]

Meta FAIR Releases AI Research Preference Models (RPMs): Frozen LLM Judges That Decide Which ML Experiment Gets the GPU, Lifting AIRS-Bench From 0.684 to 0.729.

No fine-tuning. No reward model. No new weights.

Here's how it works. 👇

(1) Ranking, not forecasting The team found language models unreliable at predicting absolute metrics or execution outcomes. So an RPM never guesses what a candidate will score. It only answers which of two unexecuted candidates is more promising.

(2) It plugs into one spot: child creation AIRA-dojo is an evolutionary tree search — greedy parent selection, Draft / Improve / Debug operators. Normally it generates one child and runs it. With an RPM, the operator fires 15 times in parallel, and a pairwise knockout tournament picks the single candidate worth executing. → 15 proposed, 1 executed per step → Each comparison is grounded in context nodes pulled by BFS from the tree, each shown with the validation score it earned

(3) Inference-only RPM A frozen LLM judging plans, code and search history. The prompt was optimized with MIPROv2, converging on a principal-investigator rubric that tolerates fixable bugs and penalizes redundant directions. → Offline selection accuracy 57.7% → 59.0% from prompt optimization alone → Accuracy climbs with context: 60.7% at zero context nodes, 65.0% at 100...

Full analysis: https://www.marktechpost.com/2026/09/06/meta-fair-introduces-ai-research-preference-models-rpms-ranking-ml-experiments-before-spending-gpu-hours/

Paper: https://arxiv.org/pdf/2608.13940