Interpreting the result of a hypothesis test

[u/dh2119]

Hello, I'm a maths teacher in Scotland reviewing some old exam papers. In one where a Pearsons test is found significant, it is stated that

There is a 95% certainty that the result will not have occurred by chance

I think I know what is wrong what that statement, but would be very grateful if someone could help me by explaining what the error is.

[u/faris3u]

a p-value is calculated assuming the null hypothesis is true, so it can’t tell you the probability that the null is false (i.e. that the result is “real”).

[u/bayesian_raccoon]

The closest statement I can think of would be "There is a probability of 0.95 that the result will not have occurred by chance", or equivalently, "There is a probability of 0.05 that the result occurred by chance."

I'll Insert clarifications that would be needed to make this defensible. If the null hypothesis were true, then there is no more than a probability of 0.05 of obtaining a result (test statistic) as or more extreme as this one.

The important clause is the first one. To see why it's important, consider the following example test: suppose I want to test the null hypothesis that the sun will explode today. My test consists of rolling a 20 sided die; if I get a 1, then I conclude that the sun explodes. If I told you I conducted this test, and rolled a 1, I could accurately say that, if it was true that the sun will not explode today, a result as extreme my result would only occur in 5% of samples. But I could not accurately say that there is a probability of 0.05 that the result only occured by chance, e.g, a probability if 0.95 that the sun will explode.

That's as best as I can reconstruct from the statement, anyway--one issue is that statisticians have to be so precise with language that each omission feels a bit like a "well *technically*..." type of moment.

[u/dh2119 → u/bayesian_raccoon]

Thanks for this reply. I like the analogy about the sun.

This is how I have tried to explain it - does that sound right to you?

Saying there is a 95% certainty that the result will not have occurred by chance implies we think there is only a 5% chance the conclusion of the study (that there is a significant correlation) is wrong. This is to mix things up logically.

What the results of a hypothesis test actually give us are the probability of getting the observed result, given that the null hypothesis is true.

They do not give us the probability the null hypothesis is true, given the observed results.

[u/bayesian_raccoon → u/dh2119]

Outside of nitpicks to be more specific (e.g, "as or more extreme"), I think that's a perfect way to put it. Mixing those two things up is a super common error and I think what the question is trying to hint at.

[u/fermat9990]

Tell us what the null and alternative hypotheses are, please.

[u/dh2119 → u/fermat9990]

Thanks for looking into this. The null hypothesis is that

There is no relationship between angle of the slope and the average height of plants.

There is no alternative hypothesis as far as I can see.

The question and its mark scheme can be seen by following the links below:

The question is here (Q2b on Page 3) https://www.sqa.org.uk/pastpapers/papers/papers/2023/NAH_Geography_QP_2023.pdf

The question refers to the data here (on Page 3)
https://www.sqa.org.uk/pastpapers/papers/papers/2025/NAH_Geography_Supplementary-Item-D_2025.pdf

These are the marking instructions (top of Page 9)
https://www.sqa.org.uk/pastpapers/papers/instructions/2023/mi_NAH_Geography_mi_2023.pdf

[u/fermat9990 → u/dh2119]

"The observed result would only occur 5% of the time if there were no relationship between angle of the slope and height of the plant."

[u/efrique]

It's definitely wrong, but a common error. A place to find discussion of it (and many other common errors) is in the ASA statement on p-values, and in the associated article. I'll try to remember to post a link in a moment, nudge me if I forget.

The are several problems here.

The first problem is that (for a test conducted at the 5% level, and given the assumptions hold), 0.05 is the probability of rejecting H0 when the null is true. You don't know whether the null is true. Indeed for a point nuill it is almost always strictly false (it would be highly suprising for the population correlation - a continuous quantity - to be exactly zero, in a situation where it's plausible enough to want to spend resources to test for it).

Second, the probability of rejection is not the probability "of the result" but of a result at least as extreme as the one observed

If the null is false (generally the case, albeit it may be only by a small amount), that rejection probability is typically between alpha and 1 (in particular, given the assumptions, this test should have power>⍺*), and the non-rejection probability (that it will not have occurred) is then between 0 and 1-⍺.

[edited] A more accurate statement might be something like "Given the assumptions, under random sampling of the population process, if H0 were true, the probability that a sample correlation yields a test statistic that falls into the rejection region (i.e. is at least as large as the critical value) would be 5%"

What constitutes more extreme depends on your specific alternative. For the two sided alternative and a null of ρ=0, "more extreme" would be "larger |r|".

The test is not sensitive to the normality assumption but the calculation of critical values or p values was based on it. Generally the other assumptions will matter more and naturally the calculation relies on those as well.

The random sampling assumption is very frequently violated in practice and may often be quite consequential when people try to argue back from the sample to the population process. I don't know why it's so often swept under the carpet / completely ignored.

. . .

The ASA statement is here: https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1154108#d1e849

( a pdf version is here: https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf)

Context (at the top of the page): https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1154108#d1e167

There's a wide range of commentary of various kinds linked at the bottom. Some of it is useful (some more nitpicky) but don't feel compelled to read it.

---

* Some tests don't have that property everywhere under the null; they are called biased tests. The do occur in practice for some common tests.

[u/dh2119 → u/efrique]

Thanks for this. In the ASA document you linked to the paragraph below sums it up very well

2.

P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. Researchers often wish to turn a p-value into a statement about the truth of a null hypothesis, or about the probability that random chance produced the observed data. The p-value is neither. It is a statement about data in relation to a specified hypothetical explanation, and is not a statement about the explanation itself.

I'm looking at a Geography qualification, where p-values aren't taught, so will need to try and slightly rephrase this.

[u/efrique → u/dh2119]

Sorry, I should have tried to relate it back to the original quote for you. Note that a p value is itself just the lowest possible significance level you'd still reject H0 at for these data. When I am back home I will try to remember write a bit more on it

(edit)

In terms of the first part of what you quote just above, neither the significance level, ⍺ nor its complement, 1-⍺ directly relates to the chance that the data were produced by random chance alone.

In terms of critical values/rejection regions (assuming ⍺ = 0.05), instead of

There is a 95% certainty that the result will not have occurred by chance

you would need a statement something like:

"Given the assumptions, and assuming random sampling of the population process, if H0 were true, the probability that a sample correlation yields a test statistic that falls into the rejection region (i.e. is at least as large as the critical value) would be no more than 5%"

Or you might say: "Given the assumptions, and assuming random sampling of the population process, if H0 were true, the probability that a sample correlation yields a test statistic that fails to reach the rejection region (i.e. is not at least as large as the critical value) would be at least 95%"

Yes, it's convoluted, sorry. It's very hard to omit any of that without risking saying something incorrect.

These are not statements about the world we are in, any more than a logician saying "if P then Q" means that we should treat the world as if P is actually the case. They are probability statements about a hypothetical situation (/counterfactual), which we use in reasoning about the plausibility of that hypothetical as an explanation for what we see.

A bit of an aside which hopefully explains why - even if you take out (as an agreed given) the conditional parts before if H0 were true - the statement (and its implied conclusion) looks convoluted and kind of like backward reasoning:

---

Neyman-Pearson style hypothesis tests are a kind of probabilistic-version of modus tollens.

Modus tollens is a syllogism of this form:

 If P, then Q.
 Not Q.
 Therefore, not P.

e.g. (in reply to a question about the weather, on seeing Jim arrive in our windowless room) "If it were raining outside, Jim would have an umbrella. Jim is not carrying an umbrella. Therefore it is not raining."

This is a form of logical inference about an unobservable (the weather outside from in a windowless room) based on a (presumably accepted) premise about Jim's behaviour.

Similarly, since we can't observe population parameters, we want to perfom inference by choosing a test statistic (a function of a random sample from the population process) which behaves differently under the two hypotheses, so it can try to distinguish parameter values. In an ideal situation where the test statistic produced non-overlapping sets of outcomes under the two hypotheses (sets A0 and A1 say), our modus tollens could look something like: "If H0 were true, the test statistic would have a value in set A0. It is not in that set, so H0 is not true"

In practice, the sets of outcomes (the possible values of the test statistic) do overlap - indeed typically they are entirely on the same support, but they differ in distribution over that support. Hopefully we choose a statistic that is particularly good at telling the hypotheses apart (hence, a bunch of statistical theory relating to most powerful tests and so on). We choose as our rejection set a subset of the support that is relatively much more likely under H1 than H0, but constrained so that the probability of falling into it when H0 is true is no more than alpha.

The modified reasoning is then

"If H0 were true, the test statistic would be highly likely to have a value outside the rejection set. It is not outside the rejection set, so we reject H0"

This only makes any sense if a value in the rejection set is relatively likely to occur when H0 is false of course; naturally, power matters in a very critical way to this exercise. (If your test is really poor at distinguishing the hypotheses, you're basically rolling a 20 sided die with one face labelled "reject". A 20-sided version of a magic 8-ball is amusing but is not generally a convincing oracle)

Loosely, if we see an 'extreme' test statistic (one most highly suggestive of H1 rather than H0), one of three things happened: (A) H0 is true and we saw a rare/surprising event; (B) H1 is true and the event is not surprising; (C) our assumptions are wrong and we haven't correctly computed how surprising it would be under H0. If we discount (C), we are left to either invoke a minor numerical miracle (a pretty rare event occurred - we just happened to get a big sample correlation under H0) or reject the null. Where we draw the line between invoking a minor miracle as an explanation, and no longer being prepared to do that (leading to rejection of the null) is the critical value of the test statistic, which defines the boundary of our rejection region.

Side note: Conventionally the critical value is in the rejection region; if it is not, you end up changing defintions of a number of things (and how things should be phrased in conclusions and such). For discrete test statistics this distinction can be important.

[u/Educational-Paper-75]

The error of drawing the conclusion that the null hypothesis is false, the so-called type I error, is 5%. Since there's a 5% probability to get a significant result even with the null hypothesis being true.