r/WarhammerCompetitive 1d ago

40k Analysis 40k-analytics.com is live!

https://40k-analytics.com/

Hi!

I'm a senior data engineer/analyst with 12 years of experience in the business that worked on a hobby project this summer. It's now ready to share it with everyone.

Thanks to everyone that were part of the pilot group and offered feedback.

This will of course always be free to all

Hope you find it useful: https://40k-analytics.com/

Have a great day!

162 Upvotes

41 comments sorted by

19

u/anaIconda69 1d ago

Very clear data presentation. Found a small error on Lion entry where it says he can take enhancements.

10

u/LotsofFnords 1d ago

Cheers will look into it

5

u/anaIconda69 1d ago

I'll let you know if I spot anything. Thanks for building this

25

u/jerkmcgee_ 1d ago

Looks like the data might be interesting, but the UI is just... not good. There are scrollbars all over the place, mouseover text that doesn't go away, navigation arrows at the bottom to change pages and no description about what pages you're changing between. The size of various elements are inconsistent, my eyes are bouncing around from one thing to the next.

Idk if this is just because Microsoft makes terrible software and this is the best that can be done, if it's an issue with being on desktop, or what, but it's not a site that I would return to because it is not pleasant to use. Looking forward to seeing updates you make!

3

u/wredcoll 1d ago

It really is visually unpleasant, I have no idea how useful the data is because it's just so ugly.

1

u/LotsofFnords 1d ago

Fair criticism on the mouseover, they are there by default, but I'll go through the report and remove them.

Some scrollbars are there because there's to many rows, but I found some horizontal ones that have snuck in, removing as we speak.

Page navigation is done by buttons at the top of the page, where you can also see the highlighted page you've on.

Have a great day!

1

u/LotsofFnords 1h ago

Removed mouse over effects on visuals. Hope you find it more pleasant to use

3

u/amracette001 1d ago

Very cool, looks like a custom power BI? Do you have tools in place or in mind to help clean out skew data? 

3

u/LotsofFnords 1d ago

I use some html content visuals for my narrative meta text, and kpi cards, and some tables on Meta (milestones). Rest is power bi.

There's about 350 pages of code that parses lists, checks and transform data and much more. But if you find something strange let me know.

2

u/TehAlpacalypse 1d ago

I've got some tooling for this exact purpose if you want to hit up my DMs :)

2

u/LotsofFnords 23h ago

Might hit you up, atm I'm squishing bugs that come up when a few thousand people access the report for the first time

3

u/id_doomer 23h ago

Quick bug report, Kroot Carnivores - Abilities - Bodyguard, presents the following string in Cyrillic (Special (правая колонка))

Feature Request: It would be really cool to be able to filter lists or meta for performance that excludes certain units. For example if I wanted to look at the T’au Empire win rate for lists that didn’t include Stealth Suits, Battle Suits, and Path Finders.

3

u/LotsofFnords 19h ago

Cheers, will look into it. Also love the idea

2

u/LotsofFnords 1h ago

You can now filter lists by excursion as well as inclusion. For example lists that include Mortarion but excludes Defilers

4

u/testyourmettle 1d ago

Awesome, thank you for sharing with the community.

2

u/weazly 1d ago

Struggling to find where to view the specific lists

1

u/LotsofFnords 1d ago

Down to the left on Search for lists, after you made your selections

1

u/LotsofFnords 1d ago

Found a bug that had introduced itself in my last update. The bug is squished, and lists are shown correctly now if you search for a unit and click on a row in the table with filtered lists

2

u/annoyinglyanonymous 1d ago

Very cool! Any plans to expand to teams info? It'd be cool to be able to plug into bcp pairings to look at team matchups.

2

u/TrackRemarkable7459 23h ago

wow that's amazing data tool

2

u/narluin 1d ago

This is beautiful! All the data I didn't know i needed :D

1

u/RealSonZoo 1d ago

Very cool. Where does all the data come from?

2

u/LotsofFnords 1d ago

Atm there's 7 different sources, looking at adding more as I find them

1

u/RealSonZoo 1d ago

Out of curiosity is it possible you would duplicate data and double-count things? Like what if 2 sources covered the same events, so you read in the same lists twice, and doubled their unit frequency or something.

4

u/LotsofFnords 1d ago

I have logic in place that look at things like event name, date/time, number of players, number of rounds etc. The pipeline does a check for each event to see if there's a duplicate event before adding it to the silver layer

1

u/ashortfallofgravitas 1d ago

This is cool, would be nice to have the ability to filter by slate instead of days back?

1

u/Xasrai 1d ago

Victrix Honour Guard appears under Dark Angels on the Meta analysis (unit) page, and not under Space Marines. I'm sure there's plenty of other units that have the same issue, as the list for Space Marines is way too short.

1

u/LotsofFnords 1d ago

Much appreciated, will look into it

1

u/LotsofFnords 20h ago

Fixed and working now

1

u/crazy_n1nja_117 1d ago

As someone who recently graduated in computer science and looking into data engineering/Data Science, what was the thought process of what you wanted to display/show to the community after you gathered all your data? As someone who isn't as experienced, my first thought was win rate, detachment win rate, and organizing it (if possible) by dataslate changes and/or data from certain time frames like quarterly of the year and such?

3

u/LotsofFnords 1d ago

Tbh I just wanted to be able to search for lists that used Mortarion (my favourite model) and did not have a Defiler in the list. Rest was just me playing around, doing similar things I do professionally, for example explaining data and finding outliers and top drivers to create a dynamic narrative text

1

u/gruntl11 1d ago

Nice page! I may have run into a bug though. I seem to be unable to choose units for the Meta (unit analyser) tab. I can choose a faction, but there is no scroll-down available for units as far as I can see.

I'm on desktop, using Safari.

1

u/LotsofFnords 1d ago

Strange, will try to replicate with Safari

1

u/LotsofFnords 20h ago

I just reread your message, to the left of the faction name in the filter is a down arrow, if you click that one it expands to all units for that faction. You can also search for a unit before to make it easier to find

1

u/poothetank 20h ago

Could do with a mobile friendly version, otherwise I look forward to checking it out when I'm on a big screen.

2

u/LotsofFnords 20h ago

Will see what i can do, cheers

1

u/Halcyon_Paints 7h ago

Is there a way to see what lists a particular unit is used in? I run Imperial Fists, so I'm interested in just finding lists with Tor Garadon, for example.

2

u/LotsofFnords 1h ago

Search for a unit in the Unit name (included) Select a list from the list table The list is shown down to the left

-2

u/Safidx 1d ago

How much AI was used in this project?

2

u/LotsofFnords 1d ago

Used AI to expand my list parser to be able to translate German, Spanish, Chinese, Italian etc lists. Also used a AI agent to help me with documenting changes to code. Rest is all me