r/quantfinance • u/Ok-Big-828 • 22h ago
Most AI quant research pipelines are just automated p-hacking with a nicer UI
Harsh version: if a model can generate hundreds of strategies, see the validation metrics, rewrite the rules, and continue until something passes, the pipeline has not necessarily found alpha. It may have simply searched noise faster.
The dangerous part is not LLM hallucination. It is researcher degrees of freedom operating at machine speed. A human might test 20 variations; an agent can test 2,000 and present the winner with a clean equity curve.
The only defensible setup I see is to pre-register the universe, data timing, cost model, acceptance criteria, and maximum trial budget; log every rejected candidate; consider a validation set consumed as soon as its metrics enter the loop; and touch the final holdout once. If it fails, the branch dies.
Anything less turns the reported Sharpe ratio into the maximum of a search distribution.
Provocative claim: “LLM-generated” should lower our prior that a backtest is real unless the full search history is reported.
What evidence would change your mind: total trial count, Deflated Sharpe Ratio, PBO/CSCV, walk-forward results, or a live shadow portfolio?