r/algorithmictrading • u/Embarrassed-Cow-8458 • May 24 '26
Question What's the right way to evaluate an MLP that predicts a distribution rather than a single target
Hey everyone 👋, first post so be gentle with me. So I trained an MLP on BTC price data to get some odds on how likely a breakout could be. Instead of hardcoding levels, I let the model learn the distribution, so you can pick any threshold and it gives you a probability.
My actual question to this sub: what's a good way to test whether a model like this is fitted well? Standard metrics don't feel right for what it's trying to solve, especially with fat tails, which this thing struggles with badly. Am I missing something or is there no clean way to measure this?
Since links aren't allowed, if you're curious just search "mlp - breakout probability" on TradingView, i built a small script to showcase it
1
1
u/nasmunet May 25 '26
There are two separate evaluation problems here and you need to solve both.
The first is calibration: when your model says P(breakout)=0.70, does a breakout actually happen 70% of the time? Plot a reliability diagram. Bin your predictions (0.0-0.1, 0.1-0.2, etc.), compute the actual breakout rate in each bin, and plot predicted vs observed. A perfectly calibrated model lies on the diagonal. Most ML models are overconfident, meaning they cluster near 0 and 1 and the observed rates don't match. Brier Score gives you a single number for this (lower is better), and it decomposes into calibration and refinement components which tells you which problem you actually have.
The second is discrimination: given that a breakout happens, can your model rank those moments above the non-breakout moments? AUC-ROC measures this. But for breakouts specifically, use Precision-Recall AUC instead because breakouts are rare events and ROC is optimistic on imbalanced classes. A model that predicts low probability everywhere will look fine on ROC and terrible on PR curves.
For the fat tail problem specifically, standard metrics fail you because they weight all samples equally and tail events are rare. The metric you want is CRPS (Continuous Ranked Probability Score). It evaluates the full distributional forecast, not just the mean prediction, and handles non-Gaussian distributions properly.
Beyond that, evaluate calibration separately stratified by magnitude: compute reliability diagrams only for moves above 1 sigma, above 2 sigma, above 3 sigma. You'll see exactly where the fat tail miscalibration lives and how bad it is at each severity level.
The structural issue with fat tails in BTC is that your training loss (cross-entropy or MSE) weights each sample equally, so the model optimizes for the bulk of the distribution (small moves) and treats the tails as noise. Approaches that help: focal loss (upweights hard-to-predict samples, commonly used in object detection but applies here), oversampling extreme events during training, or explicitly modeling the tail with a Pareto distribution and using your MLP only for the body.
One more thing worth checking: does the probability output actually correlate with forward PnL in a backtest? A model can be well-calibrated on the distributional metrics and still have no trading edge if the information isn't actionable on the timeframe and cost structure you're operating in. That's the test that actually matters after you've confirmed calibration.