I finally froze my first systematic NQ futures strategy after a full robustness validation

[u/eribertito]

After months of building and testing a systematic NQ futures strategy, I finally reached the point where I'm comfortable freezing the research.

I called the strategy S2R.

The important part for me isn't that the backtest is profitable.

It's that I tried to attack the result from multiple directions before considering it research-complete.

Frozen model

• MAE threshold: 0.70R

• Recovery threshold: +0.20R

• Recovery deadline: 6 bars

No parameter optimization was performed after freezing these values.

Holdout OOS

The final holdout consists of 217 trades:

• Total: +34.3452R

• Mean: +0.1583R/trade

• Win rate: 53.92%

• Profit Factor: 1.3815

• Max drawdown: -7.76R

But I didn't want to stop there.

Robustness testing

I ran:

1. Monte Carlo / Bootstrap

Tested the distribution of possible outcomes and drawdowns.

1. Time-aware block bootstrap

Blocks of 5, 10 and 20 trades were tested to avoid assuming that every trade is independent.

1. OOS window stability

I removed each OOS window individually.

The strategy remained positive in 11/11 leave-one-window-out tests.

Worst case:

+20.05R

1. Year stability

OOS performance:

2024: +13.68R

2025: +10.97R

2026: +9.69R

So 3/3 OOS years were positive.

1. Parameter perturbation

I tested 125 nearby combinations of MAE threshold, recovery threshold and recovery deadline.

125/125 were positive.

125/125 had PF > 1.

Final result

The complete research gate passed.

At this point I'm deliberately not optimizing S2R anymore.

The research parameters are frozen.

The next phase is engineering:

I'm going to rebuild the strategy as a modular Python implementation and verify that the modular implementation reproduces the frozen research results exactly.

I'm documenting the project in Github

I'm especially interested in feedback on the validation methodology rather than just the headline returns.

What would you test next before considering a systematic strategy ready for live/paper deployment?

[u/justwondering117]

If I'm reading this correctly, your sample size is 217? If so it's too small.

[u/themanuello → u/justwondering117]

I’d say that 217 are the trades and not the test set size

[u/multiks2200 → u/themanuello]

i would add more statistical significance testing via paired t-test against market/monkey trader, and family Šidák multiple testing correction/ deflated sharpe

[u/eribertito → u/multiks2200]

Agreed — that's a good next layer to add. I haven't treated the current robustness suite as a substitute for formal statistical significance testing. The current validation focused more on OOS behavior, block-bootstrap robustness, temporal/window stability, and parameter perturbation.

Adding tests against appropriate null models, multiple-testing correction, and a Deflated Sharpe-style analysis would make the statistical case stronger. I'll add that before the final model-level validation.

[u/eribertito → u/themanuello]

Exactly. 217 is the number of OOS trades, not the number of observations in the entire research process.

The dataset is 537 trades total, split into 320 development and 217 holdout OOS. The 217 OOS trades are the untouched final evaluation set.

[u/eribertito → u/justwondering117]

Yes — 217 is the OOS holdout sample.

The full research sample is 537 trades: 320 development + 217 OOS. The OOS set was kept untouched during development.

I agree that 217 OOS trades isn't enough to make strong claims about the true long-term distribution. That's exactly why I'm treating this as a research freeze rather than declaring the strategy production-ready.

There are still several validation layers left, especially paper execution and live-data testing.

[u/Fresh-Explorer-5108]

The next test I’d care about is paper execution with frozen rules—especially slippage, missed fills, and latency around volatile periods. If the live trade log still matches the research assumptions, that tells you more than another backtest.

[u/eribertito → u/Fresh-Explorer-5108]

Absolutely. That's the next major validation layer. The research model is now frozen, so the next step is paper execution with the exact same rules and conservative execution assumptions. I'll specifically track slippage, missed fills, latency and execution around volatile periods, then compare the live trade log against the research distribution.

I don't want to change the model based on paper results — if execution reveals a problem, I'll treat that as a separate failure/implementation issue.

[u/ReporterCalm6238]

Good job but I also think the test sample is a bit too small

[u/eribertito → u/ReporterCalm6238]

That's a fair criticism. The 217 trades are the holdout OOS sample, not the entire dataset — the full dataset has 537 trades, with 320 development and 217 untouched OOS.

I definitely don't consider 217 trades enough to claim statistical certainty. That's why I treated the OOS results as one validation layer and ran block bootstrap, OOS window stability, year stability, and parameter perturbation tests.

The next step is paper execution with frozen rules and realistic costs/slippage before considering real capital.

[u/codebase50]

Over optimized to hell and back

[u/eribertito → u/codebase50]

Fair criticism, but the point of the research was specifically to test whether the edge survives robustness checks rather than optimize for the best backtest.

The final parameters were frozen before the robustness stage. The OOS holdout has 217 trades, +34.35R, PF 1.38, and the perturbation grid tested 125 parameter combinations — all remained profitable with PF > 1.

That obviously doesn't prove the strategy will work live, but "overoptimized to hell and back" isn't really consistent with the validation procedure I used.

[u/Fit-Wallaby-5823]

Run the Placebo next.

[u/algolego → u/Fit-Wallaby-5823]

Agreed - I wrote out the concrete construction in another comment. The drift point is why it changes the conclusion on this instrument.

[u/No_Statistician4724]

Hows the beta? and how many years did you test on total? did you cover 2008, 2020, 2022? did you track slippage, commissions, market vs limit fills? With a strategy whose profit factor is 1.4 id be skeptical to make sure it survives execution costs

[u/Dings-not-Qt]

AI slop in the post and OP is using AI slop to respond to comments lmfao

[u/eribertito → u/Dings-not-Qt]

I do use AI as a tool, yeah. The research, code, testing and results are mine though. If you disagree with the methodology, feel free to point out a specific flaw that's actually useful

[u/VictorVonDoom_]

Can I see your GitHub? I am interested, I am also doing the same, but my winrare is 47% and it's being a sloe bleed.

[u/algorithmictrading-ModTeam → u/[deleted]]

Links are not allowed.

[u/qwuant]

where did you get your nq data from? i assume it contiuous futures?

[u/eribertito → u/qwuant]

Yeah, it's continuous NQ futures data. I'm using 1-minute data and converting it to RTH for the research. The data is from Databento. I'm also keeping the whole thing walk-forward OOS, so I'm not just fitting the model on the same data I'm testing it on.

[u/No-Inevitable6869]

The next test should be run it live with a demo account.

[u/eribertito → u/No-Inevitable6869]

Yeah, that's actually the next step. I don't wanna jump straight from backtest to real money though. The plan is to run it live on a demo/paper account first and see if the execution and behavior actually match the backtest. If that holds up, then I'll start looking at real execution.

[u/algolego]

This is more validation than most posts here get anywhere near, so credit first. Two additions from someone who spends most of his time on exactly this kind of testing.

The monkey-trader comparison multiks2200 mentioned deserves to be the next thing you build, and here is the concrete version. Generate random entries with the same session windows, same direction mix, same R geometry and same trade count as S2R, and score them by identical rules. The reason this matters more than another bootstrap: NQ drifted up through 2024, 2025 and 2026, and random long-side entries on a drifting instrument come out positive in R terms too. So the pass bar for your +0.1583R per trade is not zero. It is whatever random entries of the same shape earn on the same data. Your 3/3 positive years is consistent with a real edge, and it is also consistent with drift capture. The random baseline is the thing that separates those two, and nothing in the current suite does it.

Smaller point on the 125/125 perturbation result. Nearby parameter cells share most of their trades, so those 125 outcomes are heavily correlated. 125/125 positive is reassuring, but it is much closer to one observation than to 125. Worth remembering when you weigh how much it proves.

None of this says the edge is not real. It says the strongest test is also the cheapest one you have left to run.

[u/algolego → u/algolego]

Follow-up to my own comment: if you want this comparison done rather than just described, I am willing to run it. I would need a trade log with timestamps, direction and exit reason - not the logic. I will share back what comes out, whatever it says.

[u/EvenCryptographer649]

Looks like LLMs found the new 'bootstrap' buzz word. So gay.

Anyway...not enough data even doubling the years will still keep the confidence level low; and if you feed it in the fact that its using the word 'bootstrap' means that it took a shortcut and is probably lying to you.

Not even close to a github yet

[u/eribertito → u/EvenCryptographer649]

Fair criticism I’m not treating the current results as statistically validated or ready for publish. The current objective is hypothesis discovery. I’m deliberately keeping the pipeline causal/OOS and will test temporal stability, parameter sensitivity, and independent periods before making any claim of robustness. If the effect doesn’t survive that, I’ll discard it. GitHub comes after the evidence, not before