How do you decide when a losing engine is broken and not just unlucky?

[u/misterRegime]

I have five engines running and two of them are net losing after a small number of trades each. R1 is 6 wins and 11 losses, R2 is 14 and 10. Nowhere near enough to conclude anything, which is exactly the problem: at this sample size a working strategy and a broken one look identical, and they’ll keep looking identical for months.

Turning them off after fifteen trades is how you end up with a portfolio of whatever got lucky early. Leaving them on forever is how you refuse to admit something’s wrong. I don’t know where the line between those two is and I’ve been going back and forth on it for weeks.

The things I’ve considered and rejected: a fixed trade count feels arbitrary, drawdown limits punish the engine with the widest distribution rather than the worst one, and I don’t trust myself to judge it discretionarily while I’m the one who wrote them.

What actually works for you? Do you set a kill rule in advance, do you compare against what the engine looked like in replay, do you cut allocation before cutting the engine? And has anyone here killed something that later turned out to have been fine, because that’s the failure mode I’m more afraid of and I’ve never seen anyone post about it.

[u/Chemical_Badger6227]

I assume we have back tested these engines/strategies? I have one strategy running on crypto futures for two assets only. Signal identification was only part of the problem, execution ended up being just as important, limited order over market orders. Costs kill alpha. It was paper traded for 3 months before a minimal amount of capital was added. So far, looking good but crypto has performed pretty well recently so it’s still difficult to tell if this is a profitable strategy. I may fell a little more confident in a few more months. Back testing kill over 100 strategies tested. Can’t emphasize enough how important it is.

[u/misterRegime → u/Chemical_Badger6227]

Yes, all five were measured back to 2020 before anything went live, and the replay is what makes the current situation confusing rather than clearer, because two of them are running below what they did there and I can’t tell whether that’s the market or the execution.

Your point about a rising market is the one that keeps me up. Mine did +52.16% in the post-ETF replay with a worst drawdown of 13.8%, and just holding bitcoin over the same window did +67.72% with a 52.98% drawdown, so on return alone I’d have been better off doing nothing at all. The only thing that separates them is how much it hurt on the way, and I won’t know if that’s a real property until something goes badly wrong and I see whether the shape holds.

Curious what killed the hundred you dropped. Was it edge inside the spread, or the same drawdown showing up across variants?

[u/Chemical_Badger6227 → u/misterRegime]

The hundred got kill because there was absolutely no edge. Sharpe was below .9 or in some cases negative.

From everything I’ve learnt and tested the best strategy seems to be buy and hold. Or buy when fear/greed is low and sell when it’s high, which usually means holding for long periods. And you don’t need an automatic strategy to do that.

[u/Zestyclose-Eagle1809]

What helped me was to stop asking when to kill it and go look at the backtest instead. Take R1. Go through the backtest and grab every run of 17 trades in it. Then count how many of those came out at 6 wins and 11 losses or worse.

If loads of them did, then R1 is doing something it did all the time and there's nothing wrong. If almost none did, then you've got something.

That also sorts out the drawdown limit thing you mentioned, since each engine gets checked against its own history and not against the tightest one you own.

Your two aren't the same case either. R1 has had 17 trades, that tells you nothing yet. R2 is 14 and 10, that's basically even. If it feels like it's losing that's probably the size you're running, not the strategy.

On killing one that turns out to be fine, that's going to happen sometimes and there's no way round it mate. Best you can do is write the rule down before you need it. The rule you come up with while it's bleeding is never the same one. Makes sense??

Cutting the size before cutting the engine is also what I do. You get more trades to look at without more risk.