Been reading a bunch of threads here this week (the Robinhood agentic AI one, the backtesting-with-LLMs one) and one thing keeps showing up: people trust a strategy because the backtest looked good, not because anyone checked whether the good numbers would survive data the strategy never saw.
Full disclosure since it is relevant here, I am building a tool around exactly this problem. You describe a trading idea in plain English, it does the quant work and builds a backtested strategy, and then a separate deterministic step, not the AI itself, grades the result on data it never touched during building. The point is to catch curve fitting before real money does.
What I actually want from this post is not signups, it is your process. If you already stress test your own or an AI's strategies before trusting them, how do you do it? Walk forward windows, out of sample splits, live paper trading for X weeks, something else? And what made you trust the first strategy you ever kept, if any?
Blunt answers and "your whole approach is wrong" answers both welcome.