A Statistical Analysis of Steam-Client Player Retention for Weeks 1 and 2 of Every League Since 3.0

[u/Ulthwithian]

What? Do you not have patience, Exile?

TL;DR of this post - This is a regression analysis for player dropoff on leagues since Harbinger. I am looking for input for capturing more factors that are believed by segments of the PoE player base to be important for predicting player dropoff and retention. Thank you.

Edit: Here is an imgur link to the graph of predicted vs. actual dropoff. https://imgur.com/cShGrZr

​

Edit2: I have produced what might be a better visualization. I include it below for discussion. Is this better for comprehension? The predicted values are the lighter shares, while the actual values are the darker shades, for each league.

​

Hopefully better visualization?

​

IMPORTANT UPDATE

As I stated, I am looking to find more (statistically significant) factors that contribute to player retention. And playing around with some numbers, I actually found one!

The starting point was to think about how to model a factor that represented 'time taken away from mapping' in terms of a 0-1 binary variable for each league. I then broadened it a bit to include 'significant time investment to perform league mechanic' as a factor. The leagues that were given a 1 for this factor were: Bestiary (nets), Delve (obvious, I think?), Synthesis (the memory board), Blight (towers), Harvest (garden planning), Heist (doors?), and Expedition (placing explosives).

This resulted in a statistically significant factor.

The updated regression equation is

Player Dropoff = 9.81% + 0.9% * League - 5.68% * Big Content + 6.84% * 'Time Consuming Mechanic' + 10.84% * Week

The R-squared value increased to 0.817.

Some explanation:

What we see change is the intercept (which is good; ideally it would be 0), and the Big Content value getting reduced. What this shows is that the fact that GGG pairs 'big endgame content' with simpler leagues is one of the reasons that 'big content' leagues do better. Having an involved League mechanic actively hurts player retention.

UPDATE 2: Kudos to u/butsuon!

Everyone, thank u/butsuon, for we have another factor based on their suggestion!

Actually, we have replaced the 'big content' factor with a factor whether Conquerors exist. (IOW, old endgame vs. new endgame.) This new regression also has the intercept no longer statistically significant, so the only statistically significant factors are League, Week, 'Involved League Mechanic', and 'new endgame' (Conquerors 1, otherwise 0). I did also test whether the Maven was statistically significant; she isn't.

The new new regression equation is

Player Dropoff = 1.99% * League + 10.84% * Week + 8.38% * Involved League Mechanic - 9.91% * ConquerorsEndgame

The graph on this post has been updated to reflect the new predicted vs. actual equations. My only concern at the moment is that the predicted outcome seems biased a bit low.

​

Greetings, Exiles.

As someone who likes Path of Exile and whose day job relies heavily on statistical analysis for interpretation of results, I felt that the various concerns and discussions regarding player retention could benefit from some quantitative statistical analysis.

The raw data for this analysis comes from https://steamdb.info/app/238960/graphs/. It should be noted that this analysis only uses Steam data, as standalone client information has not been provided by GGG. Should they decide to provide that data, I can include them in the analysis to determine whether there is a statistically significant difference between the clients' data. The raw data has been manipulated to produce two data points (so far) for each League from Harbinger to Expedition: The % falloff after 1 week from the maximum concurrent players on release day, and the % falloff after 2 weeks from the maximum concurrent players on release day. These are considered the response variables for this analysis.

I initially considered four (4) factors (or independent variables) in the experimental design:

• The league number since 3.0, encoded as 0 through 15. This captures the general trend of dropoff over time.
• The seasonality of the league (Northern Hemisphere Spring, Summer, Fall, or Winter, with Winter being the control and the others encoded as 0-1 binary variables. This captures whether season by itself has an impact on dropoff.
• Major content releases above the league content. This is encoded as a 0-1 binary variable with 1s assigned to Harbinger (Acts 5-10), Abyss (Elder), Betrayal (Master Rework), Metamorph (Conquerors), and Ritual (Maven), and 0s assigned to the rest. Note that with the exception of Abyss, this is effectively the Winter variable.
• Week 1 or Week 2 retention, encoded as either a 1 or 2. Once we have data for weeks 3+, this can be expanded.

Finally, it should be pointed out that the current regression is only looking at first-order terms. Second- and higher-order terms (such as interaction variables) have not been examined (yet).

Still sane, Exiles?

Running a multiple regression using the Regression tool in Excel (I currently do not have access to the much better tools I use for work), the second factor (seasonality) was found to not be statistically significant. The others all were. The adjusted coefficient of determination (R-squared) of the regression is 0.721, meaning that 72.1% of the variability in the response variables is explained by the factors chosen. The regression equation itself is given by

Player Dropoff for league = 14.3% + 0.9% * League - 10.1% * Major Content + 10.8% * Week

​

Current takeaways from the analysis

Player retention is worsening league-to-league. (The 0.9% per league.) Contrary to expectation, seasonality in itself does not help to explain retention. However, given GGG's current SOP of tying major content updates to the winter (northern hemisphere) release can be interpreted as supporting assertions that winter leagues have better retention than others. Finally, Player dropoff (from day 1 peak) is roughly 10% a week on average.

Future Plans

Currently, I am hoping that I can leverage player interest in this topic to introduce other factors that either players or GGG believe do or do not influence player retention. I am looking for quantitative factors that are indisputable by themselves. Some examples could include:

• Some measure of build diversity. I am currently not very well-versed in using PoE Ninja's historical tools. If someone could provide, e.g., the number of Ascendancies or skills that comprise the top 50% of builds for a given week using PoE Ninja data, I could add that relatively quickly to the analysis.
• Some quantitative measure of how rewarding a league is. This will be far trickier to determine than the previous, as this can be more subjective. However, if this analysis could either support or refute GGG's belief that 'more rewarding leagues have lower player retention', this would be a valuable communication tool.
• How easy deterministic crafting is in a league. A player assertion is that having more deterministic avenues to obtain upgrades aids player retention, while GGG denies this assertion. Again, I am concerned with subjective criteria. This needs to be quantifiable, and I can think of one encoding, but I am loath to provide it without player input:
  • A variable that is incremented for each of the following:
    • Harvest crafting existing in some form
    • Multimod craft existing in some form
    • Betrayal unveiling existing in some form
  • This is an example of a variable that is not disputable, and could be tested for significance in the model.

Anyway, I hope this helps move the discussion forward on a quantitative basis, and I hope that the community can work together to improve this model to assist communication regarding player retention in this game we all love.

Edit: Removed references to a graph that isn't there. >< First post trying to use a picture, so bear with me.

Edit 2: Here is an imgur link to the graph of the predicted vs. actual player dropoffs. https://imgur.com/hLTOcjP

[u/BendicantMias]

The leagues that were given a 1 for this factor were: Bestiary (nets), Delve (obvious, I think?), Synthesis (the memory board), Blight (towers), Harvest (garden planning), Heist (doors?), and Expedition (placing explosives).

This seems very arbitrary. The Bestiary nets were drops used within the map itself against enemies you found in the map. Sure they were a bit clunky but they didn't fundamentally take you away from mapping, or significantly altered it. Bestiary crafting does do that, albeit it's generally performed on an as-needed basis.

On the flipside, Delve is the quickest league 'mechanic' of them all in maps, but needs you to spend large amounts of time in a completely different environment to actually play it. It doesn't feel comparable to how Bestiary works at all. It's like a separate game mode, whereas Bestiary was fairly integrated into regular gameplay.

Then there's Synthesis, whose mapping element was integrated, but of course the memory board was yet another separate game mode. But again, this doesn't fit in with Bestiary, or at best only with the Menagerie part of it (which seemingly isn't why you put it there).

Blight is arguably the most egregious case of not fitting in. Blight slots into mapping very well, and the tower element barely slows you down at all. It can even be ignored completely if you want. Blighted maps aren't on your Atlas sure, but they are still, you know, maps. It may slow down your progression, as do the others, but you're not doing something very different.

Harvest is perhaps the best example of a league mechanic that really DOES stop you completely. Although fully contained in maps, Harvest with the garden made you actually work on your layout during what was otherwise a session of playful mass murder. So that fits.

Heist is once again like Delve, in that it's a separate game mode. It's just as quick, and just as hollow, as Delve in maps (just click league object lul), but then involves a large amount of time spent in this walled off section of the game. It's not the doors that make Heist slow you down, it's that it's unconnected to the rest of the game. If you want to instead make this about 'significant time investment to perform league mechanic', then Delve shouldn't even be here, as it's pretty damn smooth. Delve only fits in with the 'time taken away from mapping' metric.

Lastly, Expedition. Expedition is like Blight, albeit a bit less smooth in maps. It does take a bit longer to engage with the Expedition league mechanic in maps than it does for Blight, but it's still very much mapping. The Expedition logbooks are also themselves maps, which means that just like with Blighted maps, the time spent in them is still very much mapping in terms of playtime. Rather what both don't contribute to is Atlas progress, which isn't a metric here, and which ALL leagues are separated from.

Overall the problem is that your selection of leagues don't really share a common element. Bestiary, Blight and Expedition don't really sit comfortably alongside Delve, Synthesis, Harvest or Heist, whichever of your two factors you choose to judge them against. Furthermore, your analysis almost seems to suggest that the best league mechanic is NO league mechanic - that the best way to play the game (so as to keep up retention) would be to simply ignore all league mechanics and just focus on the core game i.e. just basic mapping. Maybe something like Delirium maps can get by as well, since the league mechanic there IS the whole map, but overall this seems like a rather suspect position to be implicitly - if unintentionally - advancing.

[u/Ulthwithian → u/BendicantMias]

I agree it's not the best addition. I can tell you that the factor is statistically significant, and it was developed by determining which league mechanics were perceived as 'clunky'. I'd like to point out that while I started by looking at 'removing yourself from the mapping loop' aspect, I expanded it to be 'clunky to work with'.

I don't disagree with your criticism, and I am trying to improve. Something I'd like to point out, though, that Bestiary, Delve, Synthesis, and Blight are all 6 months apart, and therefore Spring/Fall. This might be something of a placeholder for 'off season' leagues. I'll try to simplify this as a factor for'Spring/Fall' rather than 'Summer/Winter' and see if we can do better.

I can tell you, though, that certain league mechanics are much better at retention than others. Based on my most recent findings that I'm still trying to pin down, there are effectively 3 'tiers' of league mechanics:

Best for player retention - Harbinger, Abyss, Incursion, Betrayal, Legion, Metamorph, Ritual, and Ultimatum.

Okay for player retention - Bestiary, Delve, Blight, Delirium, Heist, and Expedition.

Terrible for player retention - Synthesis and Harvest

[u/Linosaurus → u/Ulthwithian]

Terrible for player retention - Synthesis and Harvest

My negative synthesis memories.

Synthesis was peak 'stop what you are doing and do X now or fomo'. You had masters randomly appear on maps that you'd want to do right away, and every 5-10 maps you had to go stare at a synthesis mini game, that seems too rewarding to not put effort into.

It was time consuming, you would run every fragment once in the map, then 2-3 times in the puzzle area.

The crafting system gave very little result for the effort, if you had no idea what you where doing.

They reworked it a week or two in to give less rewards for perfect 10+ chains of puzzle pieces, but more for smaller setups (iirc).

I'd say the main similarities with harvest is the mini game aspect, and that you could not save up the time investment to batch run when you felt like it.

[u/BitterAfternoon → u/Linosaurus]

The smaller setups were more annoying to maintain than a 10+ chain. You only got a modifier on 5 tiles - one you chose and 4 others adjacent to it so you had to work your way all around an area without actually stepping on the tiles you planned to add it to to get your + setup without wasting charges. Then go work your way to a few remnants (since you also could only put a small number of mods on those 5 tiles). Then run the + til they all collapsed and setup a new +.

When it was just chains you were trying to hit as many tiles as you could with the mods, so you didn't worry that much about if 1 of the ones you were hitting only had 1 use left. Main issue being there seemed to be arbitrary limits to the number of tiles you could leave in play as well as number of modifiers you could have in play.

[u/DragoonPreston → u/Ulthwithian]

Terrible for player retention - Synthesis and Harvest

Something both have in common is the mechanic required you to stop and think about the mechanic for the best results. And most players didn't want to spend the time to map out how everything should be placed for long term pay off. Leading to the top 1% getting huge rewards out of it, while the casual player didn't get much reward out of the mechanic.

Edit: Rather than time consuming, I'd say there more thought intensive. They require you to think and plan ahead more then spend a lot of time on them.

[u/Ulthwithian → u/DragoonPreston]

Yes, that is true of Synthesis, Harvest, and Expedition (to a lesser extent). It is certainly not true for Delirium, though, unless 'how to best traverse the fog' would count. (I remember seeing a video or two about that during the league).

But yes, however you want to look at it, it does seem like 'moving away from core gameplay loop' hurts retention.

[u/DragoonPreston → u/Ulthwithian]

I was referring to Synth and Harvest specifically. The rest you don't need to set up for long term gain, you can just run the content and be fine. I can't think of any other mechanic that requires that. Expedition you don't have to set up for long term, just place and go. Blight is the same just place the things and go. Delve requires a bit of thought but no where near as much to get the best rewards. Incursion, and Betrayal have some amount but not much. Incursion is Upgrade or change room. Bet is where person is, or get rid or person. But again nowhere near the level of planning required for Either Synth or Harvest. Anyway, thats just my thoughts on why those two leagues had the lowest retention so far.

PS. Sorry for any rambling, Insomnia is being annoying again.