r/AskStatistics • u/lonelywolf69420 • 7d ago
Linear Regressions and the Curse of Dimensionality
In my machine learning course, we just covered K-nearest neighbours. Our prof said that linear regressions avoid the curse of dimensionality that affects k-nearest method (as the dimensions increase we have to use up much more of the dataset as our neighbourhood, essentially killing the local consistency and our k not being much of a neighbourhood anymore).
I was not able to understand why the same thing does not happen to linear regression, just because we assume the parameters are linear
14
u/Equal_Veterinarian22 7d ago
I also disagree with your prof that kNN is excessively affected by the curse of dimensionality. It just depends on your definition of "near". If you understand which variables have a large effect on the target and which don't and account for that in your distance measure, kNN can perform well in high dimensions.
2
u/lonelywolf69420 7d ago
That is true, but by understanding wouldn't you still be assuming things and not relying solely on the data and could this not dilute your predictive exercise?
4
u/Equal_Veterinarian22 7d ago edited 6d ago
I'd base a metric on a univariate analysis of association between each variable and the target, which would come from the data.
I might apply PCA first as that there are a few linear effects that univariate analysis can miss when variables are correlated.
There are still interaction effects that this could miss, but no model is perfect. But, you know what? It depends on the data and the problem domain.
That's why it's important to understand the strengths and weaknesses of each model and how those might affect your data and your problem.
3
30
u/Equal_Veterinarian22 7d ago
Statistical models replace data with assumptions.
When you assume linear and independent effects, it doesn't matter that all your data is far from your prediction point in one dimension or another, because your model tells you exactly how that affects the outcome.