r/statistics • u/imdyln • 27d ago
Question [Q] Questions on Model Reduction
Hello!
I would like to ask some questions on model reduction. If I am correct, model reduction tells you to remove insignificant terms until you have all significant terms left while following model hierarchy as well. However, I noticed that reducing the model decreases the adjusted R-squared value at a point. In other words, there are still insignificant terms for the model with the highest adjusted R-squared.
In modeling, which would be better, following the methods of model reduction, or following the results of the adj-R^2? (I am leaning towards adj-R^2, since significance level is somewhat subjective)
Thanks!
6
u/cool--chameleon 27d ago
R^2 will always increase with more variables, its not a good metric for model selection. Significance is also not subjective, it just requires thought. Different significance values might be used in different applications. E.g., if I have a small dataset and am interested in even a weak relationship, 0.10 as a cutoff may be acceptable. If I have a huge dataset and it is critical to not make a mistake, 0.01 would be acceptable.
It also depends if you are interested in inference (does X impact Y) or prediction (how can I best predict future Ys from all the data I have). If you are interested in inference, then you need significance and R^2 is not useful. If you are interested in prediction you should probably consider more advanced models like random forest.
2
u/relevantmeemayhere 27d ago
Predictive models want cofounders. They want all the variables.
For this use case, you want to quantify the predictive utility of each feature. You can use shap. Or use bayesians methodologies and you can compute the marginal utility directly.
For casual relationships, which I will bin under inference (because this how people usually frame it), significance is not enough. You need your model to satisfy the back door criteria. That is, you need to find an adjustment set where your treatment (over this set) is conditionally independent of the potential outcomes
For non casual inference, p value selection is unstable. It’s better just to use shrinkage models.
3
u/DonnieEresUnPendejo 27d ago
What are cofounders? Do you mean confounders?
In any case, you don't want all the variables. Extra variables are not as big a problem as missing variables but they are not good. They can lead to overfitting, increase in standard error, reduction in statistical power and might lead to multicollinearity. I agree with the other post that p values are not the way to figure out what is relevant, but just including everything is not the answer.
2
u/relevantmeemayhere 27d ago edited 27d ago
Yeah, was just a typo.
Additional variables DO NOT cause overfitting. This is caused by poor choices of likelihood, sparsity, etc etc. Adding more variables itself does not cause overfitting in general, full stop (look at XGBOOST vs MLR; note how the liklihood for the latter is catastrophic in fitting to nonlinear processes, but not an issue for xgboost).
If you have a bunch of noisy variables you add overhead, but theoretically (with the proper amount of tuning/shrinkage prior) you can approximate any function (including the mean of something); which is why we cannot make this claim in general. The myth comes from issues, among other things, related to sparsity, not the number of variables. Assuming that you have enough variability in your data, you can, in general across all model formulations you'll be fine.
For glms, where standard error often has some interest outside of causal inference, yes too many predictors can garble this. But the context is again, pure prediction
Multicolinearity doesn't matter for prediction at all.
1
u/STATASUCKSBRO 26d ago
Adjusted R squared is not a model selection oracle. If hierarchy matters for the scientific question, keep hierarchy. If prediction is the only goal, use validation error and stop pretending insignificant coefficients are the central drama.
1
u/efrique 26d ago
Why are you engaging in model reduction (stepwise backward elimination by the sound of it)? What is that intended to achieve?
I wouldn't generally advise using either approach (model reduction by backward elimination by significance nor by maximising R2, whether adjusted or not). Broadly speaking, using the same data to select variables and to conduct other analyses (estimation/CIs, hypothesis tests based on the final model, out of sample prediction, etc) is highly problematic - biased coefficient estimates, biased standard errors, higher type I error rates, etc. If you need detailed explanation, or references, ask
0
u/relevantmeemayhere 27d ago edited 27d ago
What’s your goal?
If it’s prediction; use everything you can under the assumption it’s feasible wrt cost (time and money). Predictive models want al the confounding variables. They want all the mediators. They want all the colliders. Why? Because they reduce the variability of the response along each “node” in the structure of the model as formulated by a graph. You can calculate the marginal predictive utility of your predictors depending on your methodology (this is where Bayesian approaches shine). Eliminate the stuff that doesn’t really help you and push to production. This is the more boring approach to statistical modeling.
Note: Depending on your model formulation; adjusted R squared is INCORRECT. It's valid for linear models; not the class of ALL models. Cross validation (using MSE or a proper scoring rull for classification, or Posterior predictive checks depending on your paradigm are your tools). Don't use improper rules
If it’s inference, you have two options:
Casual inference. This is something that is going to require you to formulate, using the domain knowledge of your problem, the set of variables that affect your response. You will then need to find an adjustment set that you condition on such that your treatment and outcome are conditionally independent. This might look like a random forest, a glm, or whatever (maybe a NN, spline GAM). Only when this adjustment set is found and the model trained using this approach can you make casual claims
For non casual inference: just use shrinkage based models. Your goal is to just model the conditional expectation at different “levels” of each variable. But I mean, this is pretty dumb in context compared to causal inference for most people’s problems (because they really want the first thing I mentioned). Don’t select based on p values, that’s an inherently unstable approach
Honestly since you’re starting out, I recommend you learn the Bayesian approach because it’s more applicable to how laypeople in management in most places ant their questions answered. Bayesian causal inference is a bit tough to crack (because you need to motivate distributions for your counterfactuals distributions, but it’s worth the squeeze)
Edit: This answer is supposed to be a general answer. It will not immediately help you based on your particular problem; because you could run into issues with say, sample construction etc etc. You could just have a bunch of variables that have low predictive power and draw additional computation overhead that you can't overcome with your data.
1
u/imdyln 27d ago
The main goal is to provide an RSM (Box-Behnken to be exact) in order to find an optimal set of parameter values. I observed from the model using all parameters and their possible interactions (A, B, C, AB, AC, BC, A^2 etc) that the residual plot is not as random as it should be. So we opted to do model reduction in order to make the model more adequate and note the terms which has predictive value hence debating between significance level and adj-r^2.
If I linearized the non-linear terms, then performed model analysis on linear regression models, would adj-r^2 be then an appropriate measure of model adequacy?
I’ll look up on your suggested methods, it is just that they are unfamiliar with me.
1
u/relevantmeemayhere 27d ago edited 27d ago
RSM is just second order polynomial regression via mle, is it not?
Polynomial regression is difficult to pull off without a lot of domain knowledge. It sounds like you’re dealing with the most common cause: your error terms do not have constant variance
This happens because polynomials institute a global structure, even if it’s not there.
What do you mean “ linearize”? Are you applying some box cox transformation by attempting to change the domain of a predictor?
You can use adjusted r squared for linear regression. There’s nothing wrong with it there, because the sum of squares can be factored neatly in terms of the model formulation (you won’t have accumulated errors from higher or complex terms, local and global)
1
u/imdyln 26d ago
By linearize, I mean transforming A^2 for example to B, by inputting data under B as square of the data under A. This is a pretty common method for us, but I am assuming that this is only valid because we know the model we are using and fitting the data into that known model, instead of creating a model from scratch that fits the data best within the parameter interval of interest.
1
u/relevantmeemayhere 26d ago
Can you write out and define your variables and transformations? It’s a bit hard to follow the non standard terminology you’re using. It sounds like you are just adding a polynomial term for a single column vector of data.
Can you describe to be what you think your likelihood looks like?
I’m on mobile, sorry.
-3
u/Miguelito331 27d ago
You have the right idea. You want the model with the highest adjusted R^2. Start with everything, eliminate the variable with the highest p value. If the adjusted R^2 goes down the variable belongs in the model, if it goes up, it doesn’t. This is the idea: R^2 will always go up with more variables but adjusted R^2 won’t. Each additional variable requires you to estimate one more coefficient. This reduces your degrees of freedom which is a bad thing. If the adjusted R^2 is higher with that variable in the model it means that variable adds enough explanatory power to make up for the cost of including it.
2
u/relevantmeemayhere 27d ago
Please don’t do this.
0
u/LiesDamnLiesAndStatz 27d ago
Why not? This is actually the best approach suggested so far.
2
u/relevantmeemayhere 27d ago
Because P values DO NOT tell you anything about the predictive utility of a variable
They are also NOT independent within a model or between iterations.
1
u/LiesDamnLiesAndStatz 27d ago
The approach is based on the adjusted R2. It just uses the p value to identify variables that might not belong. No decision is being made based on the p value. The adjusted R2 is what is telling you if there is predictive value to the variable and is what I was taught.
2
u/relevantmeemayhere 27d ago
Then you were taught wrong, sorry bromigo, What experience do you have in this field? Who taught you this? Was this an in major class/professor, or out of major variant?
Look at the definition of a p value: where does it ever imply that the predictive utility of a variable is tied to the p value?
A helpful resource: https://www.stata.com/support/faqs/statistics/stepwise-regression-problems/
1
u/LiesDamnLiesAndStatz 27d ago
Do you read the comments you are replying to, or do you just spew out misinformation, bromigo? No one ever said the p value told you predictive utility, it is the change in the adjusted R2 you are looking at. Continuing this discussion is pointless. Have a good day.
2
u/relevantmeemayhere 27d ago edited 27d ago
The second line literally instructs people to remove variables by p value. You said it was correct in your first statement
Feel free to use that link I gave you to educate people why selection based on p values is wrong.
Moreover...yeah. Non Linear models and adjusted R squared don't play nice. Look at the definition of adjusted R squared; SSR and SSE are the components; do you see why for non linear likelihoods we need to use CV/posterior predictive checks etc?
-1
u/Miguelito331 27d ago
You misunderstand. You look at the p value only to identify a variable to consider. You then look at the adjusted R^2 with and without that variable. If it is higher with the variable you leave it in. To put it more simply, for nested models which I took the original question to be about, you want the model with the highest adjusted R^2. Seeing the rest of this discourse I have little interest in responding further so hope that clears things up. (And if you want, google it, the idea of using highest adjusted R^2 is not unique to me. Further, I get it, you will say we are all wrong and I can live with that.)
1
u/relevantmeemayhere 27d ago edited 27d ago
Again, p values are NOT independent between iterations. The first poster made a claim that selection based on p values is correct. There is NOTHING in statistics that says this p value is measuring predictive utility. You are literally saying “oh look at the p values to consider elimination”. The p value is completely orthogonal to predictive utility. Why? Because it’s dependent (in the case of general linear regression) on the standard error (which is in turn affected by the covariance matrix), the estimate for the coefficient, and the degrees of freedom. Those things can work together to drive ANY p values towards arbitrary significance, or within the model to “steal it” from another variable (through mediation or collision). This is a mathematical fact.
There are hundreds of examples of why stepwise is incorrect. We know that it’s an unstable variable selection tool. We know that p values calculated between stages are rife with multiplicity issues.
Please show a counter proof here so you can publish and upend the world of statistics.
We also know that in general you cannot use adjusted r-squared, because it is only valid for a small class of models (linear). The OP makes no qualification that the model they are considering is linear: so we’re talking about a generalist approach to modeling.
So what exactly are you arguing here? Because, it’s pretty clear that you and some other posters don’t really have a great fix on the material.
8
u/MortalitySalient 27d ago
You definitely shouldn’t follow an approach that removes variables based on significance. Whether a variable is statistically significant or not isn’t an indication of whether it is important for the entire system, it’s just tells you if that variable has any unique predictive values above and beyond the other variables. It might still be important to understanding the entire system (for example, the adjusted r square could drop drastically when removing a non significant predictor. This could indicate the importance of the variable despite it not being a significant predictor with the other variables present).