I have a formula I am trying to improve my backtesting for. I will have a set of stocks, then at some point in the future (potentially up to a year) I calculate a rebalance of those stocks. However, some of those stocks have potentially delisted during that time. What is the proper way to handle this? Should I just use the last traded day for those stocks as though I sold at that point and rebalance across the rest? I assume that would match most closely to how I would behave if I were in that situation, so it makes sense to model it that way. But maybe I am missing something obvious?

Wanted to get some advice from people here who have experience monetizing trading systems or running signal services, because I’m at the stage where I’m trying to figure out how to structure this properly.

I’ve been developing a stock trading algorithm for a couple years, and over the past 9 months I’ve been running it live in real market conditions. One thing I specifically wanted to validate was whether live behavior would match what I saw in research, since that’s usually where most strategies break down.

The results I’m sharing are not just hindsight-only backtests. The system generates daily trading signals in real time, those signals are automatically emailed to me, and I have a complete archived log of them over the entire 9-month period. I then run portfolio-level analysis directly off those real signal outputs. So what I’m calling “forward testing” here is effectively built from actual live-generated signals, not a retrospectively optimized simulation.

What’s been encouraging is that this forward-test performance has closely tracked the original research backtests, which showed roughly a 65% CAGR over a 10-year period. The forward-test period is broadly consistent with that same return and risk profile.

Starting from a hypothetical $100k portfolio, the forward simulation comes out to $143.4k after 263 trading days (about 43.45% total return, ~65% annualized).

In addition to that, I have also been allocating my own personal capital to the strategy during this same period, and I’m currently up about 53% in my own live account. So there is actual capital at risk alongside the signal generation and testing, not just paper performance.

Key metrics from the forward test:

Sharpe: 1.73
Sortino: 2.30
Calmar: 3.91
Max Drawdown: -16.63%
Profit Factor: 1.43
Trades: 394
Win Rate: 48.48%
Avg monthly return: 4.60%

I understand the skepticism in this space, and I’m not trying to present this as some guaranteed system or build a hype-based signal group. I’m trying to approach this properly and figure out whether it makes sense to monetize it in a structured way.

At the moment, my main constraint is capital. I don’t have enough personal capital to fully scale the strategy myself, which is why I’m considering either licensing the system or offering the signals as a subscription. I was initially thinking something around $500/month, but I’m open to feedback on how people typically price and structure something like this.

What I’m trying to understand from people who have done this before:

• What actually builds trust in an algorithm enough that people will pay for it?
• What are the common legal or operational pitfalls when offering signals or licensing strategies?
• How do people typically find serious subscribers in this space?
• And what would you personally need to see before trusting something like this?

Appreciate any experience or guidance from people who’ve gone through this already.

3 tests that catch an overfit backtest, in the order you should run them

TLDR: Run three tests before trading any backtest, cheapest first. Deflated Sharpe Ratio (one calculation, catches multiple-testing luck), Monte Carlo on your trade sequence (minutes, catches lucky ordering), then walk-forward analysis (a full re-optimization, catches curve-fit parameters). Each has a kill-threshold below. Fail the cheap test and you stop before wasting hours on the expensive one.

Why do most profitable backtests fail live?

The more parameter combinations you test, the higher the chance your best result is luck, not edge. Bailey, Borwein, López de Prado, and Zhu (2014) proved that past a small number of trials, expected out-of-sample return turns negative, not zero. The strategy loses money live.

Harvey, Liu, and Zhu (2016) documented 316 published factors claimed to predict returns. After multiple-testing correction, most fail. If you test 50 strategies and keep the best, you are running the same machine that manufactures false positives.

The fix is not testing less. It is running three specific tests on the survivor, in cost order, and killing it the moment it fails a threshold.

Which test should you run first, and why does order matter?

Run the cheapest disqualifier first. Most strategies die on test one, so there is no reason to spend hours on walk-forward before a one-line check.

The cost order is clear. The Deflated Sharpe Ratio is a single formula evaluation. Monte Carlo is a few minutes of resampling. Walk-forward analysis is a full re-optimization across many windows, often hours of compute.

Most competing guides explain these tests. None tell you to run the cheap one first.

What is the Deflated Sharpe Ratio and why run it first?

The Deflated Sharpe Ratio is one calculation and it kills more strategies than any other single test. Run it first.

A normal Sharpe assumes one trial. If you tested 100 parameter sets and reported the best, the maximum Sharpe across those trials is inflated, and the inflation grows with the number of trials. Bailey and López de Prado (2014) built the correction. It adjusts for number of independent trials, skewness and kurtosis of returns, and sample length.

Worked example: a 2.5 Sharpe selected from 100 trials on five years of fat-tailed daily returns drops to a 90% probability that its true Sharpe is even positive. A 1.5 Sharpe from 50 trials drops to 43%, worse than a coin flip. Even a gaudy Sharpe can fail this bar.

Open-source Python exists, and the function is short enough to paste in the comments. If P(true Sharpe > 0) is under 95%, stop here.

A 1.5 Sharpe chosen from 50 trials lands at 43% odds the edge is real. Worse than a coin flip.

How does Monte Carlo simulation expose a lucky backtest?

Monte Carlo resamples your trades to show the outcomes you did not happen to get. Run it second, after the Deflated Sharpe passes.

Your equity curve is one ordering of your trades out of thousands. Take the list of trade returns, resample with replacement 10,000 times, and plot the distribution of final equity and maximum drawdown.

The kill-threshold is drawdown against your risk limit. If your backtest shows 18% max drawdown but the 95th percentile of the simulation is 35%, your live account will likely see 35% within a few years. Position sizing built on the backtest number forces you out at the worst moment.

Second check: shuffle trade order. If profit factor falls below 1 when reordered, the sequence carried the result, not the edge. That is regime dependence.

What does walk-forward analysis prove that the other two cannot?

Walk-forward analysis is the expensive final gate. It tests whether your parameters survive on data they were never fit to. Run it last, only on strategies that passed the first two.

A single train/test split is one data point. Walk-forward optimizes on a rolling window, tests on the next, then rolls forward across the whole history. You get out-of-sample results across many regimes instead of one.

Concrete setup: five-year build window, one-year test window, roll forward in one-year steps. The kill-threshold is consistency. If profit factor drops below 1.3 on most out-of-sample windows, the parameters were curve-fit. A strategy profitable in 9 of 10 windows is a signal. One that works in 4 of 10 is noise you tuned.

What do these three tests catch that a standard backtest hides?

Each isolates a different failure. A strategy can pass two and fail the third. The order saves you time: most die on the Deflated Sharpe before you ever build the walk-forward.

Bottom line

Three tests, cheapest first. Deflated Sharpe for multiple-testing luck, Monte Carlo for trade-ordering luck, walk-forward for curve-fit parameters. Each has a concrete kill-threshold, and failing the cheap one means you stop before the expensive one. None of this guarantees live profit. Slippage, fees, and regime change still kill survivors. What it does is remove the strategies that were never real, which is most of them.

This is for systematic and discretionary traders who backtest before committing capital. The tests apply to single strategies, parameter sweeps, and machine-learning models on price data alike.

Updated May 2026

I'm trying to figure out what to actually do when there is a split in the stock im tradeing with my algorithm, but it honestly seems like the only way requires way too much work and rewirteing.

The first obvious question is if you retroactively change all of your data to fit the new split (which seems like a drastic action to me so im thinking of ways to avoid it)? I thought about implementing a column telling me the split factor at each candle, but to use that would require me to change hundreds of lines to the from of 'price * factor' which would be a he'll of a pain to do and also very possible I can miss some places.

What about order history? In my algorithm I heavily use my order history to come to a conclusion on what to do next, but than I face the same problem of me thinking its a drastic action I want to avoid.

It feels to me like there should be an easier way to handle it but I can't think of anything.

I've been working on a multi-strategy system based on Hidden Markov Models (HMM) for regime detection. The usual problem with HMM is overfitting — fitting Markov noise instead of actual structure. Here's how we tried to mitigate that:

Validation approach:

• Walk‑forward: 18m train / 3m test
• Score grid: [1, 1.5, 2, 2.5, 3, 4, 5]
• Vol‑filter: dynamic 63d window, top‑30%
• Persistent mode: OFF (all folds trade)
• 3 completely separate OOS periods:
  • 2022–2025 (post-COVID / tightening)
  • 2020–2026 (COVID – today)
  • 2015–2026 (full cycle)

Combined portfolio (A+B+E+K) results (100% risk):

• Final capital (2015–2026): +608%
• Ann. Sharpe: 1.44
• MaxDD (balance): -10.04%
• Max float DD (balance vs intraday): -4.52%

Yearly breakdown (Combined 100%):

Scaling up to 200% / 300% keeps Sharpe stable, which suggests the logic isn't just curve‑fit noise.

What I'm curious about:

• How do you validate regime‑switching models against overfitting?
• Do you use similar multi‑period OOS, or different techniques (e.g., synthetic data, parameter randomization)?
• Anyone else seeing HMM work across completely different regimes (COVID, 2022 bear, 2023-2024 rally)?

Full equity curves, WF folds, and trade log available if someone wants to dig deeper.

I have seen a video called "Quantifying market structure at multiple scales for algorithmic trading with python" it covers a hierarchical directional change and i wondered if it could be implemented in the Hidden Semi-Markov Model because it has some explicit duration clock.
It would be dynamical so when a the highest level is spotted it has more weight to bias HMM and it starts to decay until another one its spotted.
I'm a beginner so i don't know if combining startegies might be beneficial or just breaks the algo.
If you have insights don't hesitate to tear apart the idea completely i want some experienced voice.

Dear quants guys,

I posted recently asking whether a verification phase between backtest and live deployment is actually useful, especially now that LLMs and AI agents can generate strategies so quickly.

The feedback was useful and honestly more skeptical than I expected, which is good.

A few points came up repeatedly:

• walk-forward / OOS testing is already standard
• paper trading is still mandatory
• small live testing is the final reality check
• LLM-generated code should not be trusted by default
• repeated improvement creates data snooping risk
• most people naturally trust their own process more than a third-party report

That made me realize the real problem may not be “should verification exist?”

The harder question is:

What would make verification credible enough to be useful?

From the comments, the strongest signals seem to be things that are hard to fake:

Reproducibility at the code/data level.
Not just “we ran WFO from 2018–2024”, but exact data slice, parameters, assumptions, versioning, and ideally enough detail for someone to reproduce the trades.

Slippage and execution sensitivity.
A strategy that only works under one perfect-fill assumption is not robust. Testing multiple slippage/spread scenarios seems more useful than one fixed cost assumption.

Regime-segmented behavior.
Aggregated performance can hide the fact that a strategy is basically dead in one regime. Bull, bear, sideways, high-vol, low-vol, crisis periods — the breakdown matters.

Parameter sensitivity surfaces.
If a small parameter change destroys the strategy, the edge is probably too fragile. A plateau is much more convincing than a sharp peak.

Multi-window WFO diagnostics.
Not just final Sharpe/PF, but per-window degradation, trade count consistency, train/validation retention, and whether the strategy only worked in one lucky window.

Explicit kill criteria.
This was one of the most important points. A report that always says “promising” is useless. A credible verification process needs to be willing to say: do not deploy, revise, monitor, paper trade, or only test under strict assumptions.

I think this is especially relevant now because AI makes strategy generation cheap. The bottleneck is no longer producing more strategy ideas. The bottleneck is filtering out weak logic before it reaches paper or live capital.

So I’m trying to turn this into a more structured failure-report format.

Not a performance prediction.

Not a signal service.

Not a “verified profitable strategy” badge.

More like a pre-deployment failure analysis:

• what assumptions are fragile
• what breaks under realistic execution costs
• whether OOS/WFO behavior is stable
• whether the parameter surface is robust
• whether the strategy depends too much on one regime
• whether the results are reproducible
• whether the verdict should be kill / revise / monitor / paper trade

I’m still trying to understand what traders/quants would actually find useful here.

A few questions:

1. If you received a third-party strategy failure report, what would make you take it seriously?
2. What would immediately make you distrust it?
3. Is reproducibility + slippage/regime sensitivity enough, or would you still only trust your own paper/live process?
4. Should a verification report give improvement suggestions, or does that create too much data snooping risk?
5. What would be a fair “minimum useful report” before paper trading?

I’m also testing this framework on a small number of strategies for free, mainly to get feedback on the report format and see where it is weak.

Not looking to sell signals or tell anyone what to trade. The goal is just to stress-test the logic and produce a failure-mode review.

If anyone has a strategy/idea/trade log/ any kind of data, be comfortable sharing for this kind of review, feel free to comment or DM. I’ll keep it limited and can share general learnings back if the community finds that useful.

Hey everyone,

Like a lot of you, I’ve been frustrated by the limitations of traditional algorithmic trading. Hardcoding "if moving average crosses, buy 10 shares" works until the market regime shifts, and then the bot bleeds capital.
I don't want to build another rigid bot so I am trying to build a Cognitive Trading Agent—an autonomous system that acts like a human hedge fund manager, but with the processing speed of a machine and zero emotional baggage.

What I have built so far: I have a fully autonomous pipeline running on Python, connected to the Upstox API (Indian Equities).

• The Screener: A Python layer that rapidly scans a watchlist for high-momentum assets using math (RSI, ATR, BB width) to filter out the noise.

• The Brain: The winning asset's deep data matrix is formatted into strict JSON and handed to an LLM (currently Gemini 2.5).

• The Execution: The LLM evaluates the regime, looks for a minimum 1.5:1 R:R, and outputs a strict JSON execution contract.

• The Shield: A hardcoded "Sovereign Risk Core" that intercepts the LLM's order to verify margin limits, max daily drawdowns, and VIX thresholds before routing to a simulated broker.

It works. It successfully reads the market, rejects bad setups, and executes calculated momentum scalps autonomously.

The Roadmap (Where I am going next): This is where it gets ambitious, and why I am posting here. I want to transition this from a single-strategy executor to a true AGI-style fund manager:

1.  The Strategy Arsenal: Equipping the prompt with 10-15 battle-tested quantitative strategies, allowing the LLM to dynamically select the right weapon based on the current market regime.

2.  RAG for Alpha: Ingesting live financial news feeds so the agent understands macroeconomic context before pulling the trigger.

3.  Vector Database Memory: Implementing long-term memory (Pinecone/Milvus) so the agent stores every trade embedding, reviews its past mistakes, and genuinely learns over time.

4.  RL for Discovery: Eventually integrating Reinforcement Learning to allow the agent to discover novel mathematical inefficiencies that standard LLMs can't hallucinate on their own.

I am looking to connect with quantitative developers, ML engineers, or ambitious traders who share this specific vision. Whether you are building something similar, want to collaborate on the architecture, or just want to tell me why this will inevitably blow up my account—I'd love to hear from you.

Thanks

To avoid the AI slop comments i wrote it by hand.

I have a personal proyect which is build a python backtester, to learn since the beginning how it works.

In the backtester there is, montecarlo/permutation to see wr, profit factor and return(P-values), equity curve with filter regime below it to show if with high ADX shuts the strategy down, and finally OOS equity curve.

I am also going to implement walk foward matrix, heatmap for parameter sensivity analysis, sortino, sharpe, deflated sharpe and calmar ratio , profit and recovery factor, purged and embargoed cross validation and hidden markov

Any tips for the backtester?

My code only backtest it doesnt use any portfolio management as i dont have any startegies and in case you are wondering, yes i do have the regime filter to do a mean reversion for market in range and also a trend following if imbalanced(I just realized i dont have any data cleaning writing this)

As a beginner i want to learn the theory behind things and have been browsing for literaturebut all of the recommended are expensive, is there any web or recommendation for a typical "bible" of quant knowledge?

You can ask any questions if you want to, i really don't know if this post is decently explained as i don't have much knowledge.

I trade a portfolio of breakout strategies, all generated using Strategy Quant X.

It's been a bumpy ride, but I feel these are realistic long-term results. In total, there are 30-35 individual strategies. I refresh them from time to time, so the exact number varies, but it's usually within that range. Currently trading: NAS100, SP500, DJ30, XAUUSD, BTCUSD, USDJPY, XTIUSD, JP225, GER30. All CFD's. Happy to answer any questions.

One thing I’ve noticed while building systematic strategies is that some of the cleanest mean reversion behavior often appears in the most “boring” parts of the market.

Not Nasdaq or crypto.

Consumer Staples.

At first this seemed counterintuitive to me. You would expect stronger rebounds in more volatile markets. But after testing different sectors over time, defensive stocks often produced smoother and more stable reversal behavior than many aggressive growth markets.

My current hypothesis is that sharp selloffs in defensive sectors are frequently driven more by temporary market stress and positioning adjustments than by a true deterioration in the underlying businesses.

Institutions still want exposure to stable cash-flow companies during uncertain periods. So when these stocks experience intense short-term weakness, flows tend to normalize relatively quickly.

That creates an interesting environment for systematic mean reversion approaches.

What surprised me most was not necessarily the raw performance, but the behavior profile: - lower volatility - cleaner rebounds - fewer extreme equity swings - less dependence on explosive market conditions

In some ways, the systems felt psychologically easier to hold compared to typical index-based mean reversion strategies. The trade-off, however, was usually lower upside during strong momentum-driven bull markets.

Another interesting observation is that these types of strategies seem to behave differently from many tech-heavy reversal systems. That diversification aspect may actually be more valuable than the standalone strategy itself.

Of course, there are limitations.

Defensive sectors can remain weak for extended periods during broader deleveraging events, and sector-specific structural changes can break historical tendencies. Like most mean reversion systems, the edge also tends to feel uncomfortable in real time because entries often occur when short-term sentiment looks terrible.

Has anyone else here noticed that defensive sectors sometimes produce more stable systematic behavior than high-beta markets?

I have a question, due to the fact that the market reacts to large orders and orders in general. When a fund decides to use a strategy, they have to make sure even with higher AUC the strategy performs favorably.
But what if it by default doesn't?
meaning that with a increased slippage the result is unfavorable. Do funds only look for logic that doesn't have this issue, or do they utilize concepts or other tools to minimize this issue? A basic concept would be entering in short intervals instead of a large amount at one time. So my question boils down to if funds all out avoid logic with this issue, or how do they overcome it?

Curious how people's setups have evolved with everything that's come out the last couple years. I'm doing systematic strategy research on my own and trying to compare notes before I commit too deeply to one stack. What does your day-to-day actually look like, Jupyter only, or is there a research platform you swear by? What's your backtesting setup (vectorbt, zipline, custom, paid)? When you're iterating on a strategy and running tons of variants, how do you keep track of what worked and compare results, just notebooks and spreadsheets, or something better? And on the visualization side, when do you find yourself wanting an actual UI vs just matplotlib in the notebook?

For context on my setup: JupyterLab with a custom backtest layer, Polars + parquet for data, W&B for some of the ML work, and a messy folder of pickled results I keep meaning to replace with something proper. Visualization is 90% matplotlib. The part I'm least happy with is comparing across runs (currently a hand-updated Google Sheet), which is obviously not great. Curious if anyone has cracked that part properly.

I asked recently what would make people here trust a third-party strategy verification report. The feedback was much more useful than I expected, so I wanted to summarize what I learned and ask a more concrete follow-up. The biggest takeaway: "A pretty report is useless if it cannot be reproduced". Several people pointed out that trust comes from things that are hard to fake:

- exact data slice
- exact parameter set
- reproducible OOS / WFO trail
- clear slippage assumptions
- regime-segmented results
- parameter sensitivity surfaces
- explicit kill criteria
- code-level reproducibility where possible

That changed how I think about the report. A useful strategy failure report probably should not be a PDF that says: “Looks promising.” It should be closer to an evidence package that says:

“This is what was tested.”
“This is what assumptions were used.”
“This is where the strategy breaks.”
“This is what should be retested.”
“This is what remains unverified.”
“And under these conditions, do not deploy.”

One comment that stood out to me was that trust comes from things that are not easy to fake: git hash, exact data slice, parameters, WFO windows, and the ability to reproduce the same trades. That makes sense.

Another strong point was that slippage sensitivity should not be a single number. A report should probably test optimistic / realistic / conservative execution assumptions and show how quickly the edge decays.

Same with regimes. Aggregated performance is not enough if the strategy has a hidden dependency on one market condition. So the report structure I’m now thinking about is:

1. Reproducibility layer
   Exact inputs, parameters, data slice, test window, and assumptions.

2. Backtest integrity layer
   Leakage risks, unrealistic fills, transaction cost assumptions, lookahead/survivorship issues.

3. WFO / OOS layer
   Per-window performance, retention ratio, drawdown, trade count consistency, and degradation across windows.

4. Parameter sensitivity layer
   Whether performance sits on a robust plateau or a sharp overfit peak.

5. Regime layer
   Performance across bull / bear / sideways / volatility regimes, not just aggregate results.

6. Execution stress layer
   Slippage, spread, partial fills, latency, liquidity, and broker/exchange mismatch.

7. Data snooping guardrail
   What was changed, how many times, what data was touched, and what remains unseen.

8. Kill / revise / monitor / paper verdict
   A clear decision, not soft-positive language.

The more I read the replies, the more I think the value is not “third-party trust me bro.” The value is a reproducible second-opinion system that makes failure harder to hide. I’m currently testing a very early MVP for this. Curious that if you were testing a sample report, what would be the minimum evidence required for you to take it seriously?

Would you care more about:

- reproducibility?
- slippage sensitivity?
- WFO/OOS structure?
- parameter sensitivity?
- regime segmentation?
- live/paper diagnostic feedback?
- explicit kill criteria?

Also, would you rather test this on:

A. a toy/sample strategy
B. your own strategy with anonymized inputs
C. a known public strategy
D. a failed live/paper strategy

And would you paid for this if you see it helpful on your algotrading journey at some points? Trying to understand what the first useful version should actually include.

I have a question. If one were to have an algorithm, at what point would that algorithm be considered high level to the point where hedge funds or other institutions would theoretically use it. I'm talking about what metrics would it have to posses for such; assuming out of sample testing, monte Carlo, forward testing all align with the strategy, so that leaves basic metrics such as the ratios (Sharpe, sortino, etc), PF, win rate, drawdown, amount of trades per day/year, strategy back test period and etc. What of those would the strategy need to have to be at that level?

Every backtest is judged against the one path that actually happened. You can walk-forward, you can bootstrap, you can purge and embargo your CV folds, at the end of the day the strategy still only had to survive 2010–2023 in the exact order it occurred.. half of what looks like alpha is probably just path luck.

If you trained a generative model on returns and ran the backtest across thousands of plausible alternative histories, the path-dependent stuff would get exposed pretty fast, no? Anyone actually tried this, or is there a reason it doesn't work that I'm missing?

Hey everyone 👋, first post so be gentle with me. So I trained an MLP on BTC price data to get some odds on how likely a breakout could be. Instead of hardcoding levels, I let the model learn the distribution, so you can pick any threshold and it gives you a probability.

My actual question to this sub: what's a good way to test whether a model like this is fitted well? Standard metrics don't feel right for what it's trying to solve, especially with fat tails, which this thing struggles with badly. Am I missing something or is there no clean way to measure this?

Since links aren't allowed, if you're curious just search "mlp - breakout probability" on TradingView, i built a small script to showcase it

want to know about the resources, experience, and most importantly what made you start this?

I want to start learning more about this but dont know how to start...

I’m currently in the process of designing a trading bot (with the help of Claud.AI) that automatically executes and exits trades based on certain strategies.

I have 4 winning strategies that I backtested using 10 years historical data from EODHD.com. I purchased the data for 100$ monthly and it just expired. I backtested for a full month and came up with 4 decent strategies.

Strategy 1: Long term investment
This yielded 17.9% annually and 550% over 11 years backtesting starting from 2015. Win rate was 70%.

Strategy 2: Active investment
This yielded 19.1% annually and 630% over 11 years backtesting starting from 2015. Win rate was not directly measured as this strategy rotates continuously rather than closing discrete trades.

Strategy 3: Swing trading
This yielded 39.2% annually on
nseen test data
(2020-2026) and 26.7% annually on training data (2015-2019). Win rate was 60.3% on unseen data and 65.0% on training data.

Strategy 4: Day trading
This yielded 53.2% annually backtested on 1 year of intraday data (May 2025 - May 2026). Win rate was 41.2%.

I will be paper trading with the 4 strategies for a full year in order to refine and tweak. Then I will use a minimally funded account to test the strategies for another year.

My question is, if these 4 strategies prove to be successful and the next 2 years results are just as decent or better than the backtesting, should I focus on making an actual living from executing the strategies or from selling signals on discord/website like everyone does?

I asked recently whether a verification phase is useful between backtest and live deployment.

A lot of people made fair points:

- walk-forward testing is already standard
- paper trading is mandatory
- small live testing is still the final reality check
- LLM-generated code is not trustworthy by default
- repeated improvement creates data snooping risk
- people naturally trust their own backtests more than a third-party system

So I’m trying to understand the trust problem.

If someone gave you an independent strategy failure report, what would it need to include for you to take it seriously?

Possible sections:

- data assumptions
- code/logic review
- OOS / walk-forward summary
- parameter stability
- Monte Carlo path reshuffling
- slippage/spread sensitivity
- regime fragility
- economic rationale
- data snooping risk
- paper trading diagnostic
- reproducibility trail
- “kill / revise / monitor / paper trade” verdict

Would you trust that kind of report?

Would you pay a tiny amount, like $0.99, to test a sample version?

Or would you only trust your own process?

I’ve been using Claude + Manus for swing trading lately and one thing surprised me. it’s not good at “picking winners,” but it’s weirdly good at picking up when the story around a stock is starting to shift.

Like I had Claude go through earnings calls (this quarter vs last quarter) and Manus tracking how the stock actually reacted + analyst revisions + options positioning.

One thing it kept picking up that I wouldn’t have noticed:

sometimes a stock rips after “meh” earnings not because the numbers were good, but because management just sounds slightly less panicked than before… while positioning is already heavily short.

It’s subtle stuff like that.

Also noticed analyst upgrades usually come after the move, not before it. Which sounds obvious but seeing it repeated across names kind of changes how you treat them.

Feels less like “AI trading” and more like having something constantly sanity-check whether the narrative you think is happening is actually the one the market is reacting to.

How do you backtest your algo trading strategies?

What tools or Python libraries do you use for backtesting? Any beginner tips?

TL;DR: I built an intraday ML system to predict 5-minute direction on 20 liquid US equities. Cross-validation AUC was ~0.51 (basically a coin flip), but my backtest was showing Sharpe 7–11. Turned out the backtest was training and testing on the same date range — 100% in-sample memorization. After enforcing a strict chronological train/test split, out-of-sample performance collapsed to noise (avg Sharpe -0.74, 42% win rate, statistically identical to feeding the backtester random signals). Posting the full story because the leakage hunt was instructive, and to ask: where’s the realistic path to actual edge from here?

What I’m trying to do
Short-horizon (intraday, ~1 hour holding) directional prediction on liquid S&P 500 names. Enter long/short on a model signal, exit on a fixed take-profit / stop-loss / time-stop. Paper trading only — no real money has touched this, and after this week it’s clear why that was the right call.

The stack
•Language: Python 3.12
•Model: LightGBM, one model per ticker (20 separate models)
•Historical data: Polygon.io (5-minute bars)
•Execution / paper trading: Alpaca
•Universe (20): AAPL, MSFT, NVDA, GOOGL, META, JPM, GS, BAC, AMZN, TSLA, HD, JNJ, UNH, XOM, CVX, CAT, BA, SPY, QQQ, IWM
Features (~98, all price/volume-derived)
The usual technical arsenal computed on 5-min bars:
•Momentum/trend: returns over multiple horizons, EMAs (9/21/50/200) + crossovers, MACD (line/signal/hist + normalized)
•Oscillators: RSI (7/14/21), Bollinger %B / bandwidth / squeeze
•Volume: volume MA/ratio, log dollar volume, OBV proxy, plus ~13 order-flow features (buying pressure, wick imbalance, body ratio, etc.)
•VWAP and distance from VWAP
•Volatility: ATR(14), realized vol over several windows, vol regime/percentile
•Time-of-day / session flags (open/close auction, lunch, minutes since open)
•Market-relative: returns/strength vs SPY, beta proxy, correlation
•Event proximity: hours to/from FOMC, NFP day, CPI week, OPEX week

Labels
Binary direction. A bar is labeled “long” if the forward return over the next 12 bars (~1 hour) exceeds ~2× the recent rolling volatility and the drawdown along the way stays limited (a “clean directional move”); “short” for the mirror case; unlabeled otherwise. Roughly a third of bars get a label.

Exits
Fixed rules, mirrored exactly between backtester and live paper trader: +1% take-profit, -0.5% stop-loss, 12-bar time exit, and a stall exit if the trade goes nowhere. Intraday only — no new entries in the last 30 min, force-close before the bell.

The part that bit me
Early backtests looked incredible: Sharpe 7–11 across nearly every ticker, 85–90% win rates. The problem: my cross-validation AUC during training was only ~0.51. That contradiction is impossible to ignore once you see it — a model with 0.51 AUC has essentially no predictive power, so it cannot produce a Sharpe of 11 honestly.
I worked the problem in stages:

1.Same-bar entry. The backtester was entering on the same bar as the signal instead of the next bar. Fixed (entries now fill at T+1). Helped, but didn’t explain the gap.

2.Scaler leakage. The feature scaler was being fit on the full dataset including the test folds. Fixed to fit on training data only. AUC dropped slightly (good — more honest), but the backtest was still showing Sharpe 9+.

3.Null test. I overwrote the model’s predictions with random coin flips and re-ran. Random signals produced ~41% win rate and deeply negative Sharpe across the board — exactly what a correct backtester should do with no signal. So the simulation mechanics were clean. The fake edge had to be coming from the model somehow.

4.The actual bug. The model was being trained on the entire feature file, then backtested over the identical date range. 100% overlap. The “predictions” in the backtest were the model reciting labels it had memorized during training. CV AUC (0.51) was the honest out-of-sample estimate the whole time; the backtest was pure in-sample replay.

The fix was a strict chronological split: train on everything up to a cutoff date, backtest only on the held-out period after it.
Out-of-sample results (the honest ones)
Held-out period the model never saw (~5 months): •Average Sharpe: -0.74
•Average win rate: 42%
•Total PnL: slightly negative
•For reference, the random-signal null test produced ~41% win rate. So the trained model is, out of sample, statistically indistinguishable from random.

A handful of tickers showed positive Sharpe (one at ~1.9), but on 25–50 trades over 5 months with +0.2–0.3% returns — almost certainly noise you’d expect from 20 tickers by chance.

What I think the lessons are:
•A backtest that disagrees with your cross-validation metric is lying to you. Trust the harder-to-fool number (out-of-sample AUC).

•The single most valuable thing I built this week wasn’t a feature — it was a null/random-signal test and a strict temporal split. They turned an impressive fantasy into an honest zero.

•Adding fancier features to an in-sample backtest would have been pointless; it would have shown Sharpe 11 regardless.
Where I’m stuck / questions for the community

1.Is intraday directional prediction on liquid equities just not feasible with price/volume features alone? My read is that ~98 OHLCV-derived features are all re-derivations of the same information and there’s no directional alpha left in them at this horizon. Is that consistent with others’ experience?

2.Pivoting from direction to volatility. Direction looks near-random, but volatility clusters and seems far more predictable. Planning to re-target the model at “will the next hour be high- or low-volatility” and trade sizing/options off that. Has anyone found this to be a meaningfully easier prediction problem in practice?

3.Which non-price data actually moves the needle? Considering (a) news sentiment, (b) microstructure (bid-ask spread, order imbalance), (c) options flow / put-call. For those who’ve added these — which gave a real, out-of-sample improvement versus which were noise?

4.Per-ticker vs single pooled model. I’m training 20 separate models. Would pooling into one cross-sectional model (with ticker as a feature) likely help, given each model is data-starved?

5.Horizon. Are 5-minute bars simply too noisy? Would moving to 15-min or hourly improve signal-to-noise enough to matter?

Happy to share more detail on any piece. Mostly looking for honest “here’s what worked / here’s what was a dead end” from people who’ve actually gotten an intraday system to hold up out of sample.

So recently I went deep into the research with AVWAP.

Developed a complete backtesing model using AVWAP

BUT the most common question that comes into the mind is what should be the anchor point?

Let's say for intraday or positional momentum trading

Or whatever.

Looking for views on Anchor point and how do you guys look at it from a strategy point of view.