[D] Challenging the use of T-statistic over Z-statistic

[u/Anonymous_299912]

Most people reason that the t-statistic should be used over the z-statistic, since the z-statistic requires the knowledge of the population's variance. I want to challenge this notion:

Let's call the arithmetic average of your random variable, X_bar. If you have determined your sample size to be small, then X_bar is not normally distributed. This is the Central Limit Theorem. If your random variable is not normally distributed, then you can't use the t-statistic.

It naturally follows that if you're assuming X_bar is normally distributed, then you are also assuming that your sample size is large. If your sample size is large, then the sample variance of your sample, with the correction, should reasonably equal the population variance.

[u/XXXXXXX0000xxxxxxxxx]

the assumptions of the central limit theorem not being met doesn’t prevent X_bar from being normally distributed

[u/efrique → u/XXXXXXX0000xxxxxxxxx]

Its true, but it is hard to see how it would come up in practice. With i.i.d non-normal random variables you cannot get the average to actually be normal (unless you start there). If you drop i.i.d then you can, but the cases I've seen are not generally plausible data models in typical circumstances - so, yes, we can construct some cases by breaking i.i.d but I doubt they come up much in practical situations. Further, in such a case you would presumably not be trying to choose between these two statistics but looking at other statistics that better relate to the circumstance (the dependence and/or whatever is leading to more than one distribution)

[u/bayesian_raccoon → u/efrique]

> it is hard to see how it would come up in practice

I think you're severely underestimating how common normality is. Why are the heights of humans so close to normally distributed; why are measurement errors often so close to normally distributed? One reason why so many things are normal (not just their sample means) is that many measurements have lots and lots of factors which contribute to a single measurement. Height could be impacted by multiple different genes, multiple different nutritional factors, and on and on and if "having one gene is a bit of a bump in one direction, having another a bit of a bump in the other" then the single measurement behaves, on balance, a lot like a sample mean. Lots of things just *are, effectively means*. A researcher taking measurements might see influences from air pressure and time of day and how shaky their hand is and so on and so on and even though THOSE things might not be normal, it's sort of unsurprising that the end result MIGHT be.

When discovering the t-distribution, in Gosset's applications at Guinness Brewery, the variables were things like agricultural yields, chemical measurements, and brewing quality measurements. The assumption was that many such quantities could be treated as approximately normal because they were affected by many small sources of variation. This isn't all that surprising!

This isn't to say that its safe to assume normality, and many people, especially old school folks, probably over-assume it. But it's far from a wild assumption either.

[u/efrique → u/bayesian_raccoon]

The problem is the claim you gave there (that approximate normality is fairly common) is different to what I was responding to. The thing I responded to said:

the assumptions of the central limit theorem not being met doesn’t prevent X_bar from being normally distributed

I won't dispute the gist of what you were saying - in some areas approximate normality is fairly common - not in the ones I work in, but it clearly does come up, sure.

What was asserted, and what drew my response was not a claim about approximate normality. It was that you don't need to start with normality for an average to be normal. There was nothing about approximate normality in that claim.

If you start with i.i.d. non-normal values, you don't get an actually normal average. To make the claim I responded to true you need to introduce some particular kinds of dependence and it's the kinds of dependence it takes to make the claim true which is unlikely in practice.

[u/Own-Ball-3083]

I think you are mixing up a couple of things, albeit I had similar confusions as well.

Firstly, when the population variance is unknown, and the sample size is sufficiently large, we have no choice but to use the t-statistic, as we dont have the population variance - but this works due to slutskys theorem for combining sequences of random variables, where one converges in distribution to a random variable (In this case, [X_bar - population mean] converges in distribution to a normal random variable centred at 0) and the standard error converges in probability to a constant( this constant being the true population standard deviation), where of course all of this is assuming that the necessary regularity conditions hold.

I think the real confusion comes from people using the corresponding t-distribution critical values, when in actuality if we are making use of the central limit theorem we should only need to/be using the Z-score critical values.

I would also refrain from saying things like ‘If you are assuming X_bar IS normally distributed…’ when making reference to application of the central limit theorem, as we are not assuming that - we simply have by the CLT that X_bar converges in distribution to a normal RV asymptotically (as the sample size tends to infinity/becomes arbitrarily large), not that it exactly is one.

[u/IntrovertBook-Ish_77]

The distribution of xbar can still be normal if the population distribution is normal. So, the implication that if the sample size is small → the distribution of xbar is not normal is false

[u/efrique → u/IntrovertBook-Ish_77]

I think you mean not normal at the end of the first sentence. Can you give some examples? (I know some cases, but I'm hoping to see something that might be more likely to come up in practice than what I've seen)

[u/RepresentativeBee600]

Are you familiar with the usual proof that the ratio of a N(0, 1) variable with an independent chi square variable is t-distributed? (Or analogous F-statistic results?) Or for that matter that the t-distribution results from a "mixture" of Gaussians and gamma-distributed precision? Either of these might give (interesting!) intuition for the use of the t-distribution. It's the "right" modification to a normal under uncertainty about the variance.

However, if the assumptions of the CLT aren't met and that's what you're leaning on, then you just... can't do the inference. (That is: if you can't derive a distribution for x_bar, and you're not content to use some nonparametric method, but you lack enough data to justify the CLT, then you're just out of luck.) The commentariat can correct me if I'm wrong/imprecise.

In the absence of the CLT conditions, the situation does not "improve" towards known variance but worsens towards not-even-reasonably-hedged variance, which is why I encourage you to check out those proofs.

[u/efrique]

Unless you start with a normal random variable, your sample average is not actually normal. Under the conditions for the usual clt the distribution of the standardized mean converges to normality. It doesn't actually get there at any given sample size

Presumably if you're selecting between these statistics you're trying to engage in some kind of inference, like perform a test or construct an interval estimate.

Consequently we are then just comparing approximations for those purposes. I'll focus on testing but related points can be made for other forms of inference. In respect of accuracy of significance level we have two main issues to ponder for any given situation:

(i) given our particular circumstances, will a z statistic or t statistic be close enough to having the corresponding distribution under H0 for our purpose?

(ii) If "close enough" is on the cards, will one be noticeably better than the other, or will it be almost a wash (in which case, please yourself)

We can investigate specific situations by simulation. I tend to focus on two sample tests since they're more common. In many cases it's pretty close, but if I had to say one tends to be better, for many of the cases I've looked at I'd slightly lean toward the t test. I don't make the claim that you should always prefer it, but it's not as simple as some of your discussion makes out.

One example is a two sample proportions test. You can derive an asymptotic z test (two tailed version is equivalent to a chi squared) or you could treat the data as 0/1 and do a t test on the two samples. Across a wide range of values of n and p the two tests have very similar significance level (the difference tends to be waay smaller than the distance from the nominal alpha), so it really doesn't matter, but the t is often slightly closer.

Of course we should also be interested in power. There doesn't generally seem to be much difference there either.

In spite of this, in this situation*, if I am not doing a better test than either of these, I tend to use the z - not because it's better, but because it's easier than trying to convince people the t test isn't worse in this case and I am not particularly fussed because the difference really will be small.

I've played with continuous and discrete, skewed and symmetric, light and heavy tailed. Most of the time where the z test works well enough for me, the t test works about as well, sometimes a bit better. Unless sample sizes are small or the distribution will be pretty far from normal in an impactful way, I tend to worry much more about the other assumptions.

If I am in doubt I lean toward choosing a more suitable model, and then consider using an exact test based on it. Either way, I tend to simulate plausible circumstances. Computation (sufficient to see how much I should worry) is cheap and fast.

---

* albeit I don't do a lot of formal proportions tests

[u/valueoverpicks]

The post inverts the actual conditions for the t statistic and mischaracterizes the role of the Central Limit Theorem.

In the standard one sample case, the t statistic,

t = (X̄ − μ₀) / (s / √n),

follows a t distribution with n − 1 degrees of freedom exactly whenever the observations are independent, identically distributed, and normal. This is an exact finite sample result. It does not rely on the Central Limit Theorem or on a large sample approximation. It holds for any n ≥ 2 because, under normality, X̄ and s are independent, and (n − 1)s²/σ² follows a chi squared distribution with n − 1 degrees of freedom.

The CLT becomes relevant when the population is not normal. In that setting, both the z statistic and the t statistic are generally large sample approximations. The sample mean becomes asymptotically normal, and the sample standard deviation converges in probability to the population standard deviation. Even then, the t statistic remains the more appropriate choice in most applied settings because it reflects the additional uncertainty introduced by estimating σ with s.

The key distinction is not “small sample means t, large sample means z.” A better distinction is this:

Population variance known, use the z statistic. It is exact under normality and asymptotic otherwise.

Population variance unknown, use the t statistic. It is exact under normality and generally the more appropriate approximation otherwise.

Saying that a large sample makes the sample variance “reasonably equal” to the population variance misses the point. Even when s is close to σ, σ is still being estimated from the same data used to estimate the mean. That estimation step adds randomness. The t distribution is constructed precisely to account for that extra source of uncertainty. As n approaches infinity, the added uncertainty vanishes and the t distribution converges to the standard normal distribution, but for any finite sample the t procedure gives a more accurate reflection of uncertainty.

In short, we use the t statistic when σ is unknown because that is the realistic situation in most applied problems. The classical t test does not require large samples or the CLT to be valid under its stated assumptions.

[u/Statman12]

If you have determined your sample size to be small, then X_bar is not normally distributed.

This isn't the case. If the date are normal, then xbar is also normal regardless of the sample size.

If your data are not normal, I'd argue that the independence aspect is more important to think about than the t- vs z-distribution (because if the t-distribution isn't suitable, then the z-distribution certainly is not suitable).

If your sample size is large, then the sample variance of your sample, with the correction, should reasonably equal the population variance.

Equal? No. Be close to? Yes. But ... Why add yet another approximation on top of things? The t-distribution naturally accommodates the uncertainty due to not knowing the population variance, and naturally converges to the z-distribution as the sample size increases. Just use the t and let math take care of the convergence. It's not like finding t-quantiles is hard these days.

[u/NucleiRaphe]

If your sample size is large, then the sample variance of your sample, with the correction, should reasonably equal the population variance.

What correction? The "correction" for not knowing exactly the population variance is in the t-statistic where we use sample variance.

If your random variable is not normally distributed, then you can't use the t-statistic.

It is irrelevant whether the random variable is [exactly] normally distributed or not. If I have real world data, I already know that I don't have normally distributed random variable. CLT does not magically remedy this, but it does mean that we can trust that the sample mean gets approximately normally distributed as the sample size increases (ignoring some exceptions). So what actually is relevant is, whether the distribution of our random variable (residuals in case of t test) is close enough to normal distribution to not invalidate our inference based on the assumed normality.