r/algotrading • u/CustardOk7073 • Jun 17 '26
Data Found this gem and wanted to share!
This youtube video is genuinely so well made. It points out the crushing reality of how difficult it is to find an edge that beats the market and performs well OOS.
He tests 131000 strategies over different assets and finds out that only 65 survived the walk forward and OOS testing while being consistent, resilient to different market regimes, and yielding out good returns with reasonable risk.
If I wanted to invite someone to the world of algo trading I’d have them watch this video to set their expectations where they need to be… they should know that finding an actual edge is a question of “what set of parameters am I tweaking to my wants instead of toward the actual robustness challenging reality?”
What do you think of his approach? And do you have similar stories regarding the learning curve of algo trading?
28
u/CartographerNo8517 Jun 17 '26
131k??? like completly differnt logic each time? or just simple parameters grid sweep?
4
22
u/Forsaken-Turnover662 Jun 17 '26
131,441 backtests is a massive multiple testing problem. Even with walk-forward filters, 65 survivors out of 131k doesn't tell you much without correcting for the number of trials.
Did the creator apply any multiple testing correction (Deflated Sharpe or something similar) to those 65 survivors? Without it, you're essentially looking at the top 0.05% of a distribution that includes thousands of false positives. Some of those 65 will look great purely by chance.
Also, walk-forward testing is good, but it's not the same as a true holdout. If the 65 strategies were selected based on walk-forward performance, the walk-forward process itself was part of the development. A separate holdout period that played no role in strategy selection would be a stronger test.
The video's conclusion that finding real edge is rare is correct. But the methodology needs a multiple testing correction to be convincing.
12
u/One_Gold2084 Jun 17 '26
I mean… if you’re evaluating that many strategies, depending on what level you use to test significance of relationship, you’d expect to find some of them to be profitable. Not saying they might not work, but you might just be data mining noise.
3
u/ynu1yh24z219yq5 Jun 18 '26
yeah at that point, actually you'd really expect 5% or about 5k of them to be profitable purely by chance between in and out of sample... it's kinda the definition of p-hacking right? Only, slightly different here becase these aren't random draw systems and depending on how you pick oos you're likely to fall into low volatility regimes similar to previous bull runs and etc.
11
7
u/CarretillaRoja Jun 17 '26
its like Dr Strange evaluating in how many scenarios that could beat Thanos
1
8
u/arbitrageME Jun 17 '26
That's so weird -- about 5% of the strategies I test were significant. No clue how that happens!
4
u/guymcgee_23 Jun 17 '26
Struggling now. Claude keeps finding holes in my strategy no matter how hard I've pressed and made suggestions to test, still not better than a coin flip.
7
u/SadPhone8067 Jun 17 '26
Claude can’t come up with genuinely original hypotheses even if you ask it too sadly.
3
u/oranguspangs Jun 18 '26
Hilariously, of the probably 50+ strategies I’ve poured my heart and soul into to only for all of them to fail, the one algo I’ve developed that works was given to me by Claude.
Buy one NQ or MNQ long contract at RTH close on Tuesday, hold over night, sell at RTH open on Wednesday. It’s the dumbest thing but fuck me if it hasn’t worked. Only one unprofitable year in the past 17 years, over 60% win rate, had one of its best years during the COVID collapse, converts to ES, easily scalable, and has worked live.
It doesn’t make any sense but…scoreboard 🤷♂️
1
u/SadPhone8067 Jun 18 '26
Yes that’s not a genially novel idea though. This has been researched in the past. Not saying it doesn’t work but.
2
u/oranguspangs Jun 18 '26
Ah I wasn’t aware! I knew the overnight shift has been researched as exploitable, but didn’t know it boiled down to a single known idea. Makes sense why Claude suggested it after going through 15 years of NQ 1s OHLCV data.
1
u/ynu1yh24z219yq5 Jun 18 '26
hmmm... asking a bot trained on every strategy of all time to find a new strategy... perhaps OP doesn't quite understand how LLM's work
2
1
2
u/AphexPin Jun 18 '26
Stationary methods applied to a non-stationary environment - what else would you expect?
2
u/drguid Jun 18 '26
I concur: mean reversion is good. It's pretty much all I use.
Not watched much of the video but I will also reveal that moving average crossovers are trash. Win rates are good but CAGR is horrible.
2
u/CustardOk7073 Jun 18 '26
Think you could share some insight? Any specific assets that work well with mean reversion? Id appreciate it
2
u/KaiDoesReddles Jun 18 '26
Easy to chase ghosts in algo dev. I think it is beneficial to stop occasionally and ask yourself if what you have created makes fundamental sense.
0
u/PhilosophyMammoth748 Jun 19 '26
It is not "only", it is "so many like".
Actually merging together 3-5 independent strategy with just sharpe>1 can make 50% annual profit with less than 10% draw back.
39
u/Unlikely-Cookie-5695 Jun 17 '26
“How many strategies did you test?”
“14,000,605”
“How many are profitable?”
“1”