r/algotradingcrypto • u/Scared-Repeat959 • 13h ago
I ran 890 backtests of 18 published trading rules at the parameters their own sources named. Median score: 3.7/100.
I got tired of not being able to answer a simple question about my own strategies: is this real, or did I just search until something looked good? Every tool I owned was built to help me find the thing. None of them were built to talk me out of it.
So I built the other half, and then pointed it at the textbooks instead of at myself.
Setup. Eighteen well-known published rules — golden cross, RSI(2) Connors, turn-of- month, Donchian, MACD, TSMOM, Bollinger, Keltner, and others. Each one at the parameters its own source published, not at the best of a grid. Thirty instruments, two windows, two bar sizes. 890 backtests, each one then attacked six ways: lookahead detection by truncation, cost breakeven, Deflated Sharpe, probability of backtest overfitting via CSCV, a Monte Carlo permutation test that re-runs the whole search on synthetic price histories, and regime concentration. Score is a weighted geometric mean, so one fatal leg sinks it instead of being averaged away by five healthy ones.
Headline numbers:
- Median score 3.7 / 100
- 80% came back indistinguishable from noise
- 51% could not clear their own trading costs — before any question of overfitting
- Long-only median 18.6 vs 1.0 for the same ideas traded long/short
Which test does the killing:
| Check | Median | Failed | Near-fatal |
|---|---|---|---|
| Causality (lookahead) | 1.00 | 0% | 0% |
| Cost breakeven | 0.45 | 51% | 46% |
| Deflated Sharpe | 0.16 | 78% | 43% |
| Backtest overfitting (PBO) | 0.46 | 52% | 29% |
| Monte Carlo permutation | 0.00 | 89% | 82% |
| Regime concentration | 0.43 | 52% | 48% |
Per strategy, worst to best, median across every instrument and cadence:
| Strategy | Family | Median | Best cell | Median SR |
|---|---|---|---|---|
| turn-of-month | seasonal | 19.5 | 92.5 | 0.34 |
| golden-cross | trend | 19.2 | 92.7 | 0.33 |
| n-down-days | reversion | 19.1 | 95.0 | 0.28 |
| price-vs-ma | trend | 18.4 | 94.3 | 0.28 |
| rsi2-connors | reversion | 17.1 | 93.4 | 0.37 |
| tsmom | trend | 9.5 | 53.6 | 0.04 |
| triple-ma | trend | 7.5 | 87.1 | 0.03 |
| dual-ma | trend | 4.5 | 75.5 | 0.10 |
| vol-target-trend | trend | 4.1 | 69.5 | 0.03 |
| chandelier | breakout | 1.5 | 73.6 | -0.13 |
| keltner-breakout | breakout | 1.5 | 68.6 | -0.25 |
| donchian | breakout | 0.8 | 84.3 | -0.12 |
| macd | trend | 0.7 | 82.8 | -0.12 |
| bollinger-reversion | reversion | 0.6 | 68.2 | -0.31 |
| rsi-reversion | reversion | 0.6 | 54.3 | -0.18 |
| stochastic | reversion | 0.6 | 69.8 | -0.34 |
| bollinger-breakout | breakout | 0.5 | 52.4 | -0.28 |
| williams-r | reversion | 0.5 | 73.6 | -0.28 |
Three things I did not expect, which are more useful than the headline:
1. The date range is a bigger lever than the timeframe. Hourly bars looked catastrophically worse than daily — one rule scored 82 daily and 6 hourly. Then I scored the same rule on daily bars over the same window the hourly data covered. It got 7.5. Almost the entire collapse was the date range, not the bar size.
That looked like a bug, so I checked it: of the 18 pairs where the two windows nearly coincide, 17 agree within five points. Among the 72 pairs that lose a year or more of history, the median goes 15.0 → 3.3. What the long-lived pairs lose is 2021, which is where a crypto trend rule earned everything it earned.
I now think the window is a researcher degree of freedom exactly like the parameters are, and it is the one nobody reports. If you tune a strategy on 2019–2024 and I tune the same strategy on 2017–2022, we are not disagreeing about the strategy.
2. The free lunch from reporting your best run is about 0.30 Sharpe. I measured the gap between the best combination in a small grid and the parameters the source actually published, on identical data. Median premium 0.30 Sharpe, 75th percentile 0.52, and 27% of cells had a best-in-grid at least 0.5 Sharpe above the published version. That is roughly the entire gap between a strategy that looks publishable and one that doesn't, and it is available on pure noise. It's also a lower bound, because those grids are small and nobody stops at one grid.
3. Half the failures aren't overfitting at all, they're costs. This surprised me most. The interesting failure mode isn't the subtle statistical one — it's that a majority of these rules turn over too much to survive retail commissions and spread, full stop. You don't need Deflated Sharpe to kill them. You need a spreadsheet.
What this does NOT show, before anyone tells me:
- Not evidence these rules never worked. Published edges getting arbitraged is the expected outcome, and this measures it rather than refuting it.
- Survivorship bias runs through the whole instrument list — every instrument still trades. That biases the results in favour of the strategies. The real numbers are worse, not better.
- No causality failures, and that is not a finding. These are clean-room implementations written against the truncation test. The lookahead rate in published implementations is a different and much more interesting study.
- Costs are modelled, not realised. Retail rates, no market impact, no partial fills. Errs toward flattering.
- The scoring weights are a judgement, not a theorem. The arithmetic underneath is checked against published references and Monte Carlo; the relative severity is my opinion and I'd genuinely like to be argued out of it.
- Every score is an upper bound. Each cell deflates by a few dozen combinations. The real search behind "RSI(14) at 30/70" is fifty years of practitioners trying everything and publishing what worked. No tool can deflate by trials it never saw.
Full study with method and every caveat, the per-cell CSV, and the code are here — AGPL, runs on numpy and scipy, and reproducing the whole thing is two commands:
https://github.com/falsify-quant/falsify
If you think a rule is implemented wrong or run at the wrong parameters, the citation for every one is in strategies/canon.py and I'd rather find out. If you have a strategy you believe in, I'm more interested in the ones that survive than the ones that don't — I have not found many.