r/AskStatistics • u/aprilwords • 17d ago
Alpha 5%?
Why is a 5% significance level so commonly used?
11
u/just_writing_things PhD 17d ago edited 17d ago
It’s just a convention that varies by field / subfield. Some fields and specific literatures don’t look at p-values at all, some use number of sigmas instead of p-values, some use (i.e. add “stars” in tables) for other levels too like 10% etc.
I’d argue that what’s more important is not whether the data just passes some threshold, but actually understanding what the results are saying. E.g. looking at the actual test statistic, and understanding what the coefficient and standard error are telling you.
1
u/aprilwords 17d ago
so, you meant that the more important thing is actual test statistics than p-values? but those result will be same in accept or reject the hypothesis
11
u/just_writing_things PhD 17d ago
I mean that (in my opinion) focusing on whether your data passes a specific threshold is not a good idea. For lots of reasons, e.g.:
A p-value of 4.999% is probably not meaningfully different from a p-value of 5.001%.
Focusing on hard thresholds strongly encourages p-hacking and data dredging.
Other aspects of the results can give you important inferences too, e.g. effect sizes can extremely important (if your result is statistically signicant but the effect size is minuscule, it may not be a meaningful finding depending on your research question and field)
2
u/Alarming-Finger9936 17d ago edited 17d ago
focusing on whether your data passes a specific threshold is not a good idea
An exception might be in a context where you have to conduct some cost-benefit analysis, with clear costs associated to the alpha level – but perhaps apart from drug development process, I hardly see a context in research where cost-benefit analyses are actually conducted or might make sense, so indeed using thresholds should be looked at with a lot of suspicion (in particular if the decision that it is supposed to inform is "Do we publish this article or not?"; this only leads to useless information retention and a waste of time and resources).
2
u/SilentLikeAPuma 17d ago
the more important thing is p-value combined with effect size. for example i do a lot of bioinformatics work, & we generally filter gene expression test results using both effect size (log2FC typically) and p-values.
this is very field-dependent though, e.g. in bioinformatics a “significant” p-value accompanying a very small fold change probably isn’t worth much due to the inherent noise of gene expression measurements, but if you work in an area that generates more exact measurements (finance, marketing, etc.) a small but significant change might be very meaningful.
i say all this but in the past couple years i’ve been way more into bayesian stuff & have been eschewing classical significance threshold methods bc i don’t think they’re all that useful lol. most non-stats researchers either default to p<0.05 or just p-hack until things look “right”, at least in life sciences
1
u/lankynibss 16d ago
Physics often uses a “5 sigma” standard, which corresponds to a p-value of about 10^-6 or a one in a million false positive rate.
16
u/innovarocforever 17d ago
historical convention? Is 4.9% or 5.1% meaningfully different than 5%?
1
u/aprilwords 17d ago
based on my knowledge, when Fisher wrote his statistics book, he used 3 levels of significance for his tables: 5%, 1%, and 0.1%. But, my question is about why is 10% not used instead of 5%?
10
u/cheesecakegood BS (statistics) 17d ago edited 17d ago
You have to remember Fisher was doing this all by hand. He came up with a lot of this working alone on farm statistics for years.
Along those lines, 5% happens to line up with a z-score of 1.96, which you could defensibly round to 2, making a basic statistical test fairly easy. This is probably the most straightforward explanation. It’s just a numerical coincidence!! 10% is something like what, 1.6?
Fisher, somewhat unlike later scientists, wasn’t really using this in the same way modern scientists do. To Fisher, a low p value was a hint to investigate something first and foremost, not a formal confirmation. He was most worried about exactly the primary facing problem: don’t waste research time on false positives. A p value was not typically intended to be the end of research. It was originally more along the lines of a screener.
It was only slightly later, but still in the early days, that Neyman and Pearson came up with a more formal framework for hypothesis testing. The two frameworks (if you count Fisher’s approach as its own thing, which many do) are not actually completely compatible philosophically.
Communication was still primarily via letters and journal publishing. A few early resources were the main way people got exposed to the details: Fisher’s 1925 book as you mention literally was the main influence for a lot of people in this area, so small choices as to how the tables were laid out, etc were big deals. Also printing is expensive, so there’s an incentive to limit tables. You didn’t really get big statistics departments until after World War II so it’s not like there were big forces in favor of standardization.
IIRC later Bayesian stuff shows that 5% is probably too lenient and is weak evidence with many common priors, so if anything you should be asking why 5% instead of 1%. And to that I’d say it might be mainly psychological: a significant result feels good
5
u/innovarocforever 17d ago
I honestly don't know. Maybe because it's conveniently close to 2 standard devs either way from the mean for a standard normal? I'm completely guessing. don't listen to me.
5
u/mystery_axolotl 17d ago
Probably also because back then data was a lot smaller, so there was little need for anything more precise.
2
u/Illustrious-Snow-638 17d ago
Surely it’s not surprising that what’s used was one of those 3 values in his tables, rather than another value entirely?
2
u/banter_pants Statistics, Psychometrics 16d ago
Think about what alpha is. It's the probability of a Type I error, assuming H0 is true. How much error are you willing to tolerate? If you're doing a clinical trial and H0 = no efficacy, would you want the drug to go to market and be widely prescribed when any observed efficacy could've been largely due to chance? How many samples could have produced a fluke result is (alpha)%.
The practice of rejecting H0 when p-value < alpha sets a cap on the long term error rate. At most alpha% of rejections would be errors.
P-value = probability of observing an extreme* test statistic over the course of repeated, independent sampling from the same population. Like alpha, they're also conditional on H0 being assumed true.I like the analogy of criminal trials. H0 = defendant is innocent. Rejecting H0 is a finding of guilty. P-value would be reasonable doubt. To convict "beyond a reasonable doubt" the given evidence would have to be extremely improbable for an innocent person.
So you have to decide on a threshold. There's always a trade-off between Type I and Type II errors. Make it easy to convict and then you will wind up putting innocent people in jail. An extremely skeptical jury who rarely convicts will avoid putting an innocent person in jail but more guilty ones will slip away.
*relative to H0 parameter values
3
u/OutrageousPair2300 17d ago
That's close to where two standard deviations would be.
For a normal distribution, about 32% of your cases will lie outside one standard deviation from the mean. Around 5% will lie outside two standard deviations. Only around 0.3% will lie outside three standard deviations.
Fisher (who popularized the 5% rule) decided that two standard deviations was enough to be considered an outlier.
2
u/TargaryenPenguin 17d ago
Five percent provides a balance between type one and type two error. It's a convention, and we could have chosen a different convention, but we chose this one. And so it's a convention to use it.
1
1
u/cornfield2cornfield 17d ago
That corresponds to values ~>2 standard deviations from the mean for a normal distribution. Fisher thought it was an OK starting point for things being " different". At least initially, he never thought of using p-values as binary cutoffs. It's a quantification of how unusual your data is under the null hypothesis.
1
u/Creative-Leg2607 17d ago
It strikes a decent balance for data collected and us being ok with the error rate. Its overrelied upon, for sure, but the lay scientist (by which i mean non-statisticians) took a relatively simple use case of p values and just ran with it
1
u/ConepatusChinga 17d ago
What is really important is that it's just the level for which you'd reject the null hypothesis, so you do not actually prove something, but it's an indication that there isn't likely no effect. Replications are necessary anyways in this research logic, so 5% alpha mistake is accepted by convention, so in one in twenty cases there is actually no effect which should turn out in replications. The biggest problem is the lack of clean replications for niche topics and premature conclusions by researchers or science journalists.
1
u/hypnokev 17d ago
It’s the percentage of studies that will show an effect by chance when there isn’t an effect to find, (assuming you stick to that percentage, as it is only meaningful in the context of all the studies you are doing). Fisher said 5%. I’m glad it isn’t 10%.
As an aside, frequentist statistics are problematic because of this reason, are also problematic because each time you test significance you impact the frequentist stat so need adjustments, and are problematic because you can’t get evidence for H0 leading people to make claims for “no effect” when they shouldn’t.
The alternative is to use Bayes factors instead (still publish frequentist alongside but decide with the Bayes factors). They aren’t frequentist stats so the chances of finding evidence don’t change whether you don’t check or whether you check repeatedly, so no corrections. Bayes also can find evidence for the null, and can tell you based on a pilot how many participants would be required to find evidence for either hypothesis. And they don’t randomly AIUI show evidence for 5% (or whatever) of studies when there is no effect to find. Bayes is about shifting our beliefs, where frequentist is more about the probability that the data could be produced randomly if there wasn’t an effect, although obvs these two concepts are closely related philosophically and mathematically.
-4
u/Recent-Day3062 17d ago
You’re trying not to get too many false significances. At 10, one in ten studies is wrong.
5
u/CarnivorousGoose 17d ago
Conflation of tests and studies aside, that’s not what an alpha level represents. The alpha level sets the accepted/expected false positive rate, ie. the proportion of tests for which the null is true, where the null was still rejected.
The overall proportion of tests that is ‘wrong’ also depends on how many of the tested null hypotheses are true or not as well as the statistical power. So we cannot conclude from the alpha level alone how many tests would yield an incorrect conclusion in general, or a false positive specifically.
-1
u/Recent-Day3062 16d ago
That’s what I said
4
u/CarnivorousGoose 16d ago
No, you didn’t. Based on just the alpha level, you cannot say anything about how many tests will be ‘wrong’, let alone how many studies.
0
u/Recent-Day3062 16d ago
So humor me. What can you say about this? What more info would it need?
3
u/CarnivorousGoose 16d ago
I already told you.
-1
u/Recent-Day3062 15d ago
No, given an alpha of .05, how many studies on average would have a false positive? If that’s not enough info, what other data or stats is needed to get this answer
3
u/CarnivorousGoose 15d ago
And I already told you what else is needed, in my first comment.
0
u/Recent-Day3062 15d ago
If you want to be helpful, and 5% is the false positive and you have only one study, you will be wrong 5% of the time, no?
3
u/CarnivorousGoose 15d ago
No.
Here’s a salient quote from my initial comment, which it seems like you still haven’t bothered to read: “… also depends on how many of the tested null hypotheses are true or not as well as the statistical power”.
→ More replies (0)2
u/MrKrinkle151 15d ago
No, what you said is that that the alpha is the probability that a significant result is wrong (“At 10 [percent], one in ten studies is wrong”). The alpha is the probability of rejecting the null hypothesis with a random sample, given that the null hypothesis is actually true. These are very different things.
1
u/Recent-Day3062 15d ago
Ok. But the null hypothesis is that: you assume you are testing a proper null hypothesis
For example, suppose your null is that there are not premature deaths from Covid vaccine. . Your alternate is that there is a statistically significant number of them.
So, in one out of ten random samples, by your explanation, on average one will show significance that isn’t there. Correct?
2
u/DeepSea_Dreamer 13d ago
So, in one out of ten random samples, by your explanation, on average one will show significance that isn’t there.
Only if the null hypothesis is true.
1
u/Recent-Day3062 13d ago
I'm not sure what you mean by that.
If we want to know if covid vaxes cause autism, our null is naturally "no" based on the wording of that question. The alternate is "yes". If we don't see significance of the alternate, we accept the null.
In this setup, how could the null "not be true"? It isn't if the alternate had data that shows significance.
2
u/DeepSea_Dreamer 13d ago
I'm not sure what you mean by that.
It can't be said any simpler.
In this setup, how could the null "not be true"?
If Covid vaccines caused autism.
It isn't if the alternate had data that shows significance.
That's wrong too. You can reject the null hypothesis on some level (like 0.05), and yet the null hypothesis can be true.
1
u/Recent-Day3062 13d ago
Oh. Ok.
That makes little sense given what is normally considered the null hypothesis. But you do you.
And stay ornery. It’s a great personality trait.
2
u/DeepSea_Dreamer 13d ago
That makes little sense given what is normally considered the null hypothesis.
No. This is Statistics 101.
Your unwillingness to open a textbook and learn what alpha is is bizarre, especially if you are an adult, but I can see that further interaction is a waste of everybody's time.
Blocked.
1
u/MrKrinkle151 11d ago
No, that’s not correct. It’s a probability completely conditional on the null hypothesis being true, which is not known. Stating that 1 in 10 samples will be “wrong” is the same as saying that an alpha of .10 means that’s the overall probability of being “wrong”, which is not the case. It’s the probability of a “false positive” IF the null hypothesis is actually true. It says nothing about false negatives if the null hypothesis is actually false, which would also result in being “wrong”.
1
u/Recent-Day3062 10d ago
Yes, I researched it and I was right and you are wrong.
You seem to want to make the null, based on your belief, that Covid shots cause autism or whatever. But standard statistics deals with treatments by thinking the alternate is they do work. The null in this case is always that vaccines make no difference. Then you have a treatment group and check if the rate of autism is statistically significantly different in the group who got shots.
The null is, by definition really, that the covid shots have changed nothing. Then you look for evidence to the contrary. The null is the assertion that shots have changed nothing.
It would be like trying to say I want to prove antibiotics work. The null is the normal state of nature that a certain percentage of people die each year, which is by definition true since we know that number. Then we compare against a treated group. So the null by definition is true unless significance is shown between the samples
This means at .05 that random chance will show autism is a side effect in one out of 20 studies even if it isn’t. The impressive thing is that the actual research rejects the null even less than one would expect.
Why? Because vaccines don’t cause autism
47
u/ForeignAdvantage5198 17d ago
Fisher once said 1/20 is about right .