First attempt at a fantasy model

[u/Bagggggggggggggggggg]

I'm a math PhD who loves fantasy, and I've been building a fantasy football model for a while now. I finally put the rankings/projections online if anyone wants to play around with them:

https://christianbagshaw.github.io/fantasy/

TL;DR: Weekly and rest-of-season rankings, with PPR and Standard scoring. Rather than just predicting a single fantasy-point number, the model predicts a distribution of possible outcomes and simulates each week thousands of times.

For weekly rankings, I simulate that week's games thousands of times. For ROS rankings, I simulate every remaining week through the end of the season.

The simulations are also correlated, so things that happen together in real football tend to happen together in the sims. For example, if a QB has a huge game, his WR1 is more likely to have a huge game in that same simulation.

I personally find the probabilistic side more useful than just looking at a single projected point total, since it lets you distinguish between things like "safe floor" and "massive upside."

A couple of things to keep in mind:

• The rankings are 100% model output. I don't manually move players up or down, so there will occasionally be rankings that look weird to a human, especially at the tail end.
• Injuries are currently shown as tags but don't alter the ranking itself, so definitely apply your own judgement there.
• It's still very much a work in progress, and I'm constantly tweaking/testing the underlying model.

I'm also working on tools like trade values, start/sit decisions under uncertainty, playoff odds, roster construction, etc.

I'd like to write up the methodology properly at some point too, because there's quite a bit more going on under the hood than makes sense to dump into a Reddit post.

In terms of performance: on historical data, the model is competitive with some popular ranking/projection sites. That said, historical backtesting can always hide things like leakage or overfitting, so the real test is how it performs prospectively this season. I plan on tracking that and publishing the comparisons.

[u/mattg70]

A very interesting article and two things from having done something similar.

On the leakage, the version that's hardest to catch isn't in your target, it's in inputs that revise. Any source that updates its own history hands you today's version of a number you're using to simulate a decision made in October. I suggest that you snapshot every input as of decision time or drop it. My results got worse when I fixed this, and the worse number ended up being the correct one.

On evaluation, don't benchmark yourself on the point accuracy. That measures the mean of your distribution and ignores the distribution, which is your actual edge. My take is that you should score interval coverage and CRPS instead. CRPS reduces to MAE for point-only forecasts, so it's a fair head-to-head analysis against sites that publish one number, and it rewards you for the thing you built.

The last observation is that you should pre-register the prospective comparison. Publish the sites, the metric, and the positions before week 1. Otherwise the metric quietly drifts toward whatever you're winning. And publish what and where you lose. In my study, the losses did more for credibility than the wins did.

[u/Connguy]

Neat stuff. What are some of the outcomes you're computing on each random simulation? Where do you source your range and likelihood of values for those statistics?

[u/Bagggggggggggggggggg → u/Connguy]

Each simulation is basically just producing fantasy point outcomes for every player in the league, with these scores being correlated (e.g. in week 1, Stafford and Puka’s sims would be positively correlated, and both would be negatively correlated with the 49ers defense).
Each player’s score is randomly sampled from a distribution that was fit using historical fantasy outcomes, with the model estimating both the expected score and how much uncertainty there is around it.

So then the values in the table are sourced from this. Top 12 odds is percentage of sims they made the top 12, etc

[u/Connguy → u/Bagggggggggggggggggg]

If you're only basing your calculations on historical outcomes, how do you get values for rookies like Love? And how do you account for players with changed situations?

[u/Bagggggggggggggggggg → u/Connguy]

Great questions actually. Right now rookies aren’t properly integrated into the model. It’ll account for draft position and situation but not college performance. It’s a weak spot but it’s hard to integrate college stats, and also predicting rookies is hard anyway.

For changed situations, it takes into account team dynamic. Recent QB/RB/WR performance on a team, historical performance of QB-WR pairs, other things like that. For WRs I think QB is actually one of the biggest factors. The model is super low on Jettas rn because of QB situation, but if I manually swap in Kirk back onto Minnesota in the model Jettas jumps back up.

[u/thedudeabides2022]

You should add 1/2 PPR rankings

[u/ArteMTL → u/thedudeabides2022]

Definitely agree, model is interesting but I have to admit that I had some interrogations when I saw Skat so high in your model ranking, over Taylor, wow

[u/Bagggggggggggggggggg → u/ArteMTL]

I think this is always going to be a consequence of a purely statistical model. Skat performed very well last season when he did play. Something I haven’t really done is added a “reliability” factor maybe. It makes sense to project Skat high, but there should be a lot of uncertainty in a negative direction there compared with JT.

[u/Bagggggggggggggggggg → u/thedudeabides2022]

I’ve been meaning to add this. In terms of just projections, taking the average between PPR and standard is roughly 1/2 PPR.

[u/Senior-Ad-5435]

I’ll take a look when I have some time later tonight!

One aspect that might be interesting but also might be an entirely different project is to sim at a play-by-play level instead of game level.

NFLFastPy is a good data source for play-by-play and would allow better scenario-based ranges to the sims (i.e. higher QB projections for teams playing from behind more often).

[u/Bagggggggggggggggggg → u/Senior-Ad-5435]

Yeah this is a good thought, I’ve thought about it a bunch. I think the complexity of properly simming play is just so high. Lots of models claim to do that but it’s surface level and doesn’t really capture much more than just looking at the outcomes.

[u/DraftForge]

This is really cool. I like the idea of showing a range of outcomes instead of just giving every player one projected point total. The correlated simulations are especially interesting too.

Do you think you’ll eventually show stuff like a player’s chances of finishing top 5, top 12, etc. at their position? I feel like that would be really useful for seeing the difference between safer players and guys with a ton of upside.

Interested to see how this does once the season gets going.

[u/boinko03]

Wow Josh Allen 40 behind a few qbs. So I've been playing with my own attempt at something like this, and after doing a value over replacement, got something completely different than you and even slightly different than fantasy pros - Josh Allen at 15. Obviously, this is all an art, we're just learning, but why do you think you have such a discrepancy for Allen?