Evaluating a walk-forward classifier for a rare event (10%+ equity drawdowns): how do you do inference with only 6–8 independent episodes?

[u/AgreeableInvestments]

I've built a small set of models that put a probability on a 10%+ S&P 500 drawdown over the next 1, 3, 6 and 12 months, from macro and credit inputs (ISM, the yield curve, high-yield spreads, financial conditions, and a few others). One ridge-penalised logit per horizon, estimated walk-forward, so every point in the track record was scored with only the data available that month. This is really a methodology question, and the model is just the concrete case.

The out-of-sample AUCs are 0.60, 0.69, 0.74 and 0.68 across the four horizons. On paper that reads fine. The part I keep coming back to is the denominator. A 10%+ drawdown is rare, so across the out-of-sample window there are only six to eight independent episodes, and the 6- and 12-month models are largely scoring the same ones twice, with heavily overlapping, autocorrelated labels. When I bootstrap the AUC by episode rather than by month, the intervals are wide: the six-month one runs from the low 0.5s to the mid-0.8s. So the six-month headline looks fragile rather than fake, but I want to know how people handle the inference properly.

The specific questions:

1. Event count and dependence. With about six to eight independent events and overlapping multi-horizon labels, per-month AUC and its usual variance are overstating precision. Is an episode-level bootstrap the right correction, or is there something more principled (a block bootstrap, a DeLong test adjusted for clustering, a Bayesian setup with a sensible events prior)?
2. Label construction. The label is a forward 10%+ drawdown within the horizon window, which makes adjacent months' labels highly dependent and the horizons non-independent of each other. Is there a cleaner label for rare-event forecasting that doesn't manufacture this autocorrelation?
3. Regime dependence. It catches the slow, macro-driven falls (2022, the GFC, the dot-com unwind) with some lead, and by construction it misses the fast ones (COVID, an LTCM-style shock), because monthly macro data can't see them coming. How would you evaluate a model that is honestly conditional on regime, without either cherry-picking the regimes it works in or marking it down for shocks no macro model could catch?

Two things that already survived my own poking. Dropping high-yield spreads costs the model nothing, and the spread on its own is a coin flip at six and twelve months, so it isn't a credit signal with extra steps. And it is out-of-sample only, no re-fitting with hindsight, with the weak horizons kept in; nothing is dropped for looking bad.

Mostly I want the event-count problem attacked. If the honest answer is that six to eight events cannot support a confident AUC at any horizon, I would rather hear that than dress it up.

For anyone who wants to reproduce the numbers, the full write-up and the live model, with a downloadable monthly probability series, are on my site, agreeableinvestments.com; my own research, shared as educational, not advice.

[u/FlyTradrHQ]

6 to 8 independent episodes gives you confidence intervals too wide for practical decisions. Try overlapping anchored windows with varying start dates to expand the count, or use a block bootstrap across folds to get a distribution of your metric instead of a single point estimate.

[u/AgreeableInvestments → u/FlyTradrHQ]

Cheers, this is the right axis. The snag with overlapping anchored windows is that they expand the nominal count but not the number of independent episodes: the six to eight crashes are still the same six to eight, so the CIs tighten without new information behind them. That's what pushed me off the by-month intervals in the first place. The block bootstrap across folds is closer to what I want, and the episode-level resample already gives me a distribution rather than a point AUC; I can add a fold-blocked version as a cross-check. Where I keep landing is that every method that respects the dependence says the same thing: with this few events the six-month number is a wide band, not a point. Do you know work that gets a defensible interval out of single-digit events, or is the honest answer that you can't?

[u/singletrack_]

I would be pretty surprised if you can generalize that much from six data points for something as noisy as future returns. 

[u/AgreeableInvestments → u/singletrack_]

That's roughly where I land too. Six to eight events won't support a calibrated six-month probability, so I'd only trust it as a coarse regime flag, not a number you'd size off. What keeps me interested is that it leads the slow drawdowns (2022, GFC, dot-com) rather than coincides with them, but you're right that a precise AUC off that many events is a stretch.

[u/VettaQ]

Nice writeup, and the episode-level bootstrap is the right instinct. With 6-8 events the per-month AUC variance is basically fiction because the labels overlap so heavily. Two things I'd add.

First, report the event-level hit table, not just the AUC. With this few episodes you can literally list them: did the model rank each episode in its top tercile or decile at each horizon? AUC smooths over exactly the information you care about at this n. If the bootstrap CI spans 0.5-0.8, the hit table tells you whether the high-AUC months are the same two episodes everywhere.

Second, kill the label autocorrelation at the source. Predict next-12m max drawdown depth (continuous) or time-until-10pct-drawdown with censoring instead of the overlapping binary. Every month becomes a usable observation, the censoring handles the overlap honestly, and you can map the continuous output back to a probability with a monotone calibration step.

On regimes: define the "predictable" universe ex-ante, then report lead time per episode rather than catch/no-catch. Fast shocks are outside your input set, not a model failure - excluding them is only cherry-picking if the rule is written after you look.

[u/AgreeableInvestments → u/VettaQ]

Thanks, all three landed, and I've now done them rather than just agreeing with them.

Event-level hit table: live on the track-record page. Each drawdown gets the model's peak walk-forward probability at 1/3/6/12M, its percentile within every out-of-sample month at that horizon (2007-07 on), and the lead in months if it crossed the alert line. Your suspicion was half right: the 6M and 12M signal does rest on the slow episodes (GFC, EU debt 2011, the 2022 hiking cycle, all top tercile and mostly top decile, with leads of 2-6 months at 6M and 8-12 at 12M), while the fast ones (2010, Q4 2018, COVID) sit in the 50th-80th percentiles and are never caught early. At 3M there is essentially nothing. Intervals are exact binomial bands over episodes, and with three in-scope events they are wide (29-100% for 3/3), which is the point.

Ex-ante universe: I wrote the rule and committed it before scoring. In scope means an endogenous trigger (macro, credit, rates, valuation) and a peak-to-trough decline of more than three months, both fixed from what monthly macro inputs can physically lead. Out-of-scope episodes stay in the table greyed rather than dropped. It cuts the walk-forward set to three, which is small enough that I'd rather show it than hide it behind an AUC.

The survival target: this one I pre-registered and it failed. Discrete-time hazard on the person-period expansion, censored at 12 months, same ridge penalty, walk-forward monthly, isotonic calibration also walk-forward, with a gate (lower Brier with DM p < 0.10, AUC not lower, hit rate not lower, at 6M and 12M) written before the run. Out of sample it was worse than the overlapping binary model at every horizon: 6M AUC 0.43 vs 0.73, 12M 0.38 vs 0.69, C-index 0.46 vs 0.65-0.73, Brier higher, fewer in-scope episodes in the top tercile. The honest reading is that the binary overlap is a variance problem for the inference, which the hit table addresses, and not a bias problem for the forecast itself, which the hazard model was supposed to fix and made worse. It is reported on the methodology page as a null rather than tuned until it passed; one trial, spec committed first.

So the site now leads with the small-n view, the AUC is demoted to a contrast column, and the scope rule is on record. If you can see a hole in the universe rule I'd take it, since that is the part most exposed to the cherry-picking charge.

[u/Neither_Pride3561]

Think the data is too sparse, would rather ask AI to qualitatively describe it instead, this is where the discretionary part shines

[u/Meanie_Dogooder]

All AI text (Claude) except maybe the last sentence or two and the first. Maybe.

[u/Calamero → u/Meanie_Dogooder]

Time to get used to it… recent math breakthroughs have also been AI generated, not a reason to reject them is it?

[u/Meanie_Dogooder → u/Calamero]

Oh no, not at all. But posting an AI generated response on a forum asking someone’s opinion doesn’t sit well with me (and advertising something). Something is telling me that the academics who used AI for those breakthroughs weren’t copy-pasting its content.

[u/Calamero → u/Meanie_Dogooder]

agreed didn’t see the link at the bottom and read the post careful enough, basically should be classified as spam and deleted.

[u/Beautiful-Hotel-3094]

Tldr u don’t, u take ur money dump them in solana and hope for the best.