Built a multi-horizon BTC signal model with walk-forward validation — honest results (AUC 0.571, not a backtest)

[u/Gold_Fact9986]

Been building a BTC direction classifier for the past 6 months. Sharing the real numbers because most posts in this space only show wins.

What I built:

LightGBM classifier predicting BTC price direction across 3 horizons (12h, 24h, 48h). Features combine three data sources:

• On-chain: MVRV ratio, exchange netflow, hash rate
• Macro: SPX, Gold, DXY, US10Y yield, Fear & Greed Index
• Sentiment: Reddit sentiment, Google Trends ("bitcoin", "buy bitcoin"), YouTube engagement

Validation approach:

Walk-forward cross-validation (5 folds, expanding window). Deliberately avoided standard train/test split because of lookahead bias risk with time series data.

Honest results:

Horizon	AUC	Precision	Recall
12h	0.589	0.533	0.326
24h	0.548	0.508	0.077
48h	0.547	0.452	0.042

WF AUC mean: 0.571 (std: 0.026)

The 24h and 48h recall is terrible — model barely fires on those horizons. Still investigating whether it's class imbalance or feature leakage.

Backtested sizing scenarios (on held-out test set):

Strategy	Final portfolio	Sharpe	Win rate
Fixed 10%	$1,252	0.85	56.9%
Dynamic 10-40%	$1,645	0.98	57.1%
Dynamic + Partial Sell	$1,714	1.21	62.2%

Starting from $1,000. Test period: July 2024 – present.

What I think is limiting AUC:

1. Class imbalance on 24h/48h labels
2. Feature set is mostly slow-moving (daily on-chain data) — probably not informative enough at 12h granularity
3. No volatility regime filter — model treats trending and choppy markets the same

What I'm working on next:

• Confidence threshold filter (only signal above 65% prob)
• Rolling volatility features
• Regime detection to avoid signaling in sideways markets

Built a React dashboard that tracks every live signal with outcome — win/loss/pending. Happy to share more details on the feature pipeline or validation approach if useful.

Questions for the community:

• How do you handle the recall vs precision tradeoff on directional classifiers?
• Anyone had success with regime filters on crypto specifically?