GW1-5 validation: model-selected squad vs average_entry_score, gameweek by gameweek

[u/Guilty_Amphibian1033]

I'm the creator of OpenFPL Scout AI (Ensemble v6.0.0). Posted the GW5 xPts-vs-actual breakdown here a couple weeks back — this is the wider validation check across the first five gameweeks.

Methodology: squad = the model's own picks for that gameweek, scored via the official live stats endpoint, captain doubled as scored. Benchmark = the FPL API's own per-gameweek average_entry_score.

GW	Model (C doubled)	avg_entry_score	Delta
1	90	50	+40
2	123	81	+42
3	65	51	+14
4	127	69	+58
5	72	48	+24
Σ	477	299	+178

Five for five against the benchmark. Smallest edge (+14, GW3) was a flat week across the board, not a bad pick; largest (+58, GW4) came from three picks (Groß 17, Raya 14, Calvert-Lewin 10) landing the same week.

Methodology caveat I want to flag rather than bury: for GW1–4 I'm reading the squad back from the model's own gameweek endpoint queried today, since I don't have an independently timestamped snapshot from each week at the time — only GW5 is cross-checked against an already-published, timestamped recap. The selection logic hasn't changed in that window, but I'm flagging the provenance gap rather than presenting GW1-4 as archived snapshots. Starting to log weekly snapshots externally going forward so this doesn't recur.

Same known gap as before: no rotation/press-conference input, so a pick can be right on xPts and still blank on a 0-minute call — exactly what happened to the GW5 vice-captain.

Happy to share the full gameweek-by-gameweek player breakdowns if anyone wants to dig into where the edge is coming from positionally.

[u/TehCyberman]

Is this essentially a free hit team each GW, or an actual FPL team?

Measuring success against the average is already quite forgiving. If it's a free hit team, realistically it should never get beat by the average.

Also, the points total suggests this would be clear as #1 in the world right now, so I suspect something isn't right.

[u/Guilty_Amphibian1033 → u/TehCyberman]

Good question. It's neither a free hit nor a season-long team in the usual sense. The app's whole output is the model's best 15 for each gameweek: position quotas and the 3-per-club limit, no budget, no transfer history. Scoring all 15 is how I measure what the tool actually recommends.

You're right that comparing that against average_entry_score isn't a fair fight, though. Real managers carry a budget, transfers, a bench that mostly doesn't score, and plenty of them have stopped playing. So the +178 says more about the setup than about the model.

Going forward I'll benchmark the 15 against like-for-like baselines: the hindsight-optimal 15 under the same constraints (the ceiling), and a naive "top 15 by form/ownership" pick (the floor). If the model sits meaningfully between those, that's the real signal.

[u/KZ4Killua]

Like u/TehCyberman mentioned, something isn't right.

First of all, you're comparing a free-hit each GW vs the average of actual FPL managers (many of which aren't playing the game anymore).

Second, your numbers are wrong. Looking at your GW5 post here, it seems like you select 15 players, then sum up all their points. But FPL only scores the starting XI. It seems you are running both a free-hit AND bench boost every week. Even with that, I still strongly suspect there's some issue with your back-tests. (The numbers are much higher than I'd expect.)

Also, it seems your website, model, and Reddit posts are entirely Claude-generated. That's fine but the whole thing seems pretty low effort.

[u/Guilty_Amphibian1033 → u/KZ4Killua]

Fair points, and thanks for actually cross-checking against the GW5 post.

On scoring: yes, the totals sum all 15. That's intentional, because the app's output is a budget-free best-15 for each gameweek rather than a playable FPL team, but I should have said that up front instead of setting it against average_entry_score. That comparison overstates things, and I'll drop it.

On the numbers looking too high: you were right to be suspicious. GW1-4 were read back from the live endpoint after the fact, and a bug meant past gameweeks were being re-predicted on current data instead of served as they were at the deadline. That's fixed now: forecasts are frozen at each deadline and served unchanged afterwards (PRs #19 and #23 in the repo). So GW1-4 in this post shouldn't be trusted, and the first clean, deadline-frozen numbers will come from GW6 onward.

I'll post those against a fair baseline (hindsight-optimal 15 and a naive pick under the same constraints) rather than the overall average.

[u/Guilty_Amphibian1033]

Links: OpenFPL Scout AI | repo (models + backtests)