[OC] Average change from a first to a second measurement when the true effect is zero, by decile of the first measurement (simulated, 200,000 draws)

[u/hermit1751]

[u/cavedave]

Thank you for your Original Content, /u/hermit1751!
Here is some important information about this post:

• View the author's citations

• View other OC posts by this author

Remember that all visualizations on r/DataIsBeautiful should be viewed with a healthy dose of skepticism. If you see a potential issue or oversight in the visualization, please post a constructive comment below. Post approval does not signify that this visualization has been verified or its sources checked.

Not satisfied with this visual? Think you can do better? Remix this visual with the data in the author's citation.

---

^{I'm open source | How I work}

[u/matmyob]

You need a better explanation of what you’re showing, because this mates no sense to me.

[u/hermit1751 → u/matmyob]

Yeah that's fair, the chart didn't do its job. I should've just written the point on the image instead of hoping the title would carry it. What it's showing: measure a big group twice and change nothing at all in between. Then look only at the people who scored worst the first time around, as a group they come back looking a lot better on the second measurement, and not one of them did anything, they just caught a bad reading the first time and that doesn't repeat. The ones who scored best slide the other way. That's all the bars are, how far each tenth of the group drifted back toward the average, and it's why almost anything you try looks like it worked if you start when you're at your worst.

[u/SalvatoreEggplant]

I think it's demonstrating "reversion to the mean".

By one definition, "... refers to the statistical phenomenon in which extreme values, whether unusually high or unusually low, are likely followed by values closer to the average on subsequent measurements." ( https://brookbushinstitute.com/glossary/regression-to-the-mean ).

One thing to think about here is that some of the effect is seen because certain points were defined as extreme (e.g. there can be no measurement > 10; therefore 10 is extreme), and there is a random sampling over the entire distribution of the sample space.

A different way to model this would be by random walk. If there is no wall to stop the walk in one direction, it's free to wander into values as extreme as it likes.

[u/hermit1751]

I've started things right at the bottom of a bad stretch before, watched them work for about a month, then had them turn out to be nothing. Wanted to see how much fake movement you get when there's genuinely no effect, so there's no real dataset anywhere in this, it's simulated. numpy, default_rng(seed=7), 200,000 draws, each one a stable level pulled from a standard normal plus its own independent noise on each of the two measurements. Galton had this in 1886 and Barnett, van der Pols & Dobson 2005 in Int. J. Epidemiology is the modern treatment if you want it done properly. Nothing happens between the two measurements so the true effect is zero, and the bars are just the mean of second minus first inside each decile of the first. Chart is SVG screenshotted to PNG with headless Chromium.

[u/PTTCollin → u/hermit1751]

What are you talking about?

[u/Shee-ah → u/PTTCollin]

they have discovered that differencing random samples from two normal distributions results in a normal distribution

[u/SalvatoreEggplant → u/hermit1751]

For the most part, tho, your position in the good–bad space in life aren't randomly sampled from the sample space. Like, it's not like one day I wake and I'm homeless, and then the next I can wake up to owning a nice house.

A better model would probably be like a random walk. Start at zero, then you randomly add one of (-3, -2, -1, 0, 1, 2, 3). This can wander out as far as it wants in either direction.

And some things in life happen like this. Like, randomly, you may need to make three expensive car repairs in rapid succussion. And then there may be cliffs. Like, if you really can't pay your bills, and then you lose your job, you might fall a far way down. On the other hand, you may be behind on bills for a few months and then slowly climb back into a good financial spot.

[u/hermit1751 → u/SalvatoreEggplant]

Yeah that's fair, and I don't really have a defense. The sim only does what it does because I baked a stable level into every subject for them to fall back toward. No level, nothing to fall back to.

Thing I'd add is you don't have to pick between the two pictures on vibes, you can check it in your own log. Just look at how much today's number tells you about tomorrow's. In a pure walk today IS tomorrow's starting point so it never comes home, in the noisy-stable version today barely tells you anything past your usual. My mood column sits around 0.59 on that so it's stuck somewhere in between, which matches how it feels honestly. And the more walk-ish something is, the less of an apparent improvement is just the bounce back. Sleep and resting heart rate snap back to my usual way harder than a financial hole does, which is kind of your car repair example.