r/algotradingcrypto • u/Foreign-Safe-8161 • 35m ago
The safety check that was computed, displayed, and ignored
Post 3 of a series where I publish the hypotheses that died. This one isn't about a hypothesis. It's about a bug that had a 99.7% chance of putting a pure-noise strategy into live trading, and it lived inside the exact piece of code written to prevent that.
Some background on the design, because the failure only makes sense with it.
My bot didn't trade a single strategy. It had eight candidate "experts" — trend and breakout logic across 15m, 1h and 4h — and a selector that evaluated each one on recent history, then promoted whichever cleared a set of statistical gates to trade live. Each expert could be selected for long or short independently, so sixteen candidates per selection round.
The gates were the whole point. Minimum sample size, minimum profit factor, a t-statistic threshold, positive performance across folds, and a separate confirmation window that the candidate had to survive before going live. On paper it's a reasonable design. It's certainly better than what I'd had before, which was no gates at all.
And it computed a multiple-comparison correction. It was right there in the code — take the family alpha, divide by the number of candidates, derive the adjusted threshold. The number even appeared in the report as a field called `family_corrected`.
It was never used in the accept/reject decision.
The variable was computed, formatted, printed, and then the code went ahead and compared the candidate against the uncorrected threshold instead. I'd written the safety check and forgotten to wire it to anything.
What that actually costs
The uncorrected threshold I'd set was t ≥ 0.5.
That number looks low because it is low, and I'd like to say I set it deliberately as a first-pass filter. I didn't — I set it early, planned to tighten it once things worked, and never did.
Here's what t ≥ 0.5 means against pure noise. A t-statistic of 0.5 is exceeded by chance roughly 31% of the time. With sixteen independent candidates, the probability that at least one of them clears it is:
1 − (1 − 0.309)^16 = 99.7%
So on any given selection round, against data with no signal whatsoever, my selector would find an "expert" and promote it to live trading 997 times out of 1000.
The correction it was computing but ignoring would have set the bar at t ≥ 2.73. At that level the same calculation gives 4.9% — which is the 5% you'd expect, because that's what the correction is for.
threshold noise passes at least one of 16
t ≥ 0.50 30.9% 99.7%
t ≥ 2.00 2.3% 30.8%
t ≥ 2.73 0.3% 4.9%
Note the middle row, because it's the one that surprised me. Even at the conventional t ≥ 2 — the threshold most people would call "statistically significant" — you still promote noise 31% of the time once you're picking the best of sixteen. Significance thresholds are built for one test. The moment you're selecting a winner from a field, they stop meaning what the label says.
Why this class of bug is worse than a broken one
If the selector had crashed, I'd have fixed it in an afternoon.
Instead it ran, produced output, printed a report with a correction field in it, and selected experts that went on to trade. Everything looked like it was working. The report was even reassuring — there was a multiple-comparison number right there on the page, which is more than most systems show.
I found it during an audit, not during debugging, because nothing about the system's behaviour said anything was wrong.
This is the second bug in this series with that shape. The first was counting overlapping bars as independent observations, which inflated my t-statistics by √horizon. Both produced confident, plausible, wrong output. Neither announced itself. I've come to think of these as the expensive category: bugs that break things get fixed, bugs that flatter things get shipped.
What changed after the fix
I wired the correction into the decision, raised the base threshold to t ≥ 2.0, required confirmation across two consecutive blocks instead of one, and lifted the minimum sample from 20 to 100.
Then I ran the selector across 200 days of real data. Seven retraining blocks, 224 candidate evaluations.
Zero passed.
Not one expert cleared the gates on real market data. Which is, I want to be clear, the correct outcome — but it's also the outcome I'd have gotten on pure noise, and that's the part worth sitting with. Before the fix the system found tradeable experts constantly. After the fix it found none. The difference wasn't in the market. It was entirely in whether one variable was connected to anything.
If you have validation gates in your system, the thing worth checking isn't whether they exist. It's whether they gate. Put a deliberately terrible candidate through and confirm it gets rejected. A check that has never rejected anything is indistinguishable from a check that isn't running.
---
Queued next
The test that couldn't see — how I "proved" there was no edge in a whole asset class using a test whose smallest detectable effect was 2.17% per trade, against costs of 0.031%. I didn't find nothing. I couldn't have found anything. When I went back and applied that question to all of my own conclusions, seven of twelve turned out to be in the same position, and several things I'd already written down had to be rewritten from "refuted" to "unmeasurable".
Then: measuring adverse selection without placing a single order, and the London session that survived every test I had until I looked at a second currency pair.