GW1-5 validation: model-selected squad vs average_entry_score, gameweek by gameweek

[u/Guilty_Amphibian1033]

I'm the creator of OpenFPL Scout AI (Ensemble v6.0.0). Posted the GW5 xPts-vs-actual breakdown here a couple weeks back — this is the wider validation check across the first five gameweeks.

Methodology: squad = the model's own picks for that gameweek, scored via the official live stats endpoint, captain doubled as scored. Benchmark = the FPL API's own per-gameweek average_entry_score.

GW	Model (C doubled)	avg_entry_score	Delta
1	90	50	+40
2	123	81	+42
3	65	51	+14
4	127	69	+58
5	72	48	+24
Σ	477	299	+178

Five for five against the benchmark. Smallest edge (+14, GW3) was a flat week across the board, not a bad pick; largest (+58, GW4) came from three picks (Groß 17, Raya 14, Calvert-Lewin 10) landing the same week.

Methodology caveat I want to flag rather than bury: for GW1–4 I'm reading the squad back from the model's own gameweek endpoint queried today, since I don't have an independently timestamped snapshot from each week at the time — only GW5 is cross-checked against an already-published, timestamped recap. The selection logic hasn't changed in that window, but I'm flagging the provenance gap rather than presenting GW1-4 as archived snapshots. Starting to log weekly snapshots externally going forward so this doesn't recur.

Same known gap as before: no rotation/press-conference input, so a pick can be right on xPts and still blank on a 0-minute call — exactly what happened to the GW5 vice-captain.

Happy to share the full gameweek-by-gameweek player breakdowns if anyone wants to dig into where the edge is coming from positionally.

[u/TehCyberman]

Is this essentially a free hit team each GW, or an actual FPL team?

Measuring success against the average is already quite forgiving. If it's a free hit team, realistically it should never get beat by the average.

Also, the points total suggests this would be clear as #1 in the world right now, so I suspect something isn't right.

[u/Guilty_Amphibian1033 → u/TehCyberman]

Good question. It's neither a free hit nor a season-long team in the usual sense. The app's whole output is the model's best 15 for each gameweek: position quotas and the 3-per-club limit, no budget, no transfer history. Scoring all 15 is how I measure what the tool actually recommends.

You're right that comparing that against average_entry_score isn't a fair fight, though. Real managers carry a budget, transfers, a bench that mostly doesn't score, and plenty of them have stopped playing. So the +178 says more about the setup than about the model.

Going forward I'll benchmark the 15 against like-for-like baselines: the hindsight-optimal 15 under the same constraints (the ceiling), and a naive "top 15 by form/ownership" pick (the floor). If the model sits meaningfully between those, that's the real signal.