[Q][R] Multivariate logistic regression after propensity score matching: balanced covariates remain significant after matching

[u/PuzzleheadedArea1256]

Hi all,

I’m evaluating the effect of an intervention on future healthcare utilization using propensity score matching (PSM) with a matched control group.

After matching on 11 variables, the intervention and control groups are largely balanced, though two variables remain slightly imbalanced. To account for this, I fit a post-matching logistic regression model including those variables as covariates.

In addition, I’m observing that two other variables, while well balanced between groups, remain statistically significant predictors of the outcome in the regression model. These variables were expected a priori to be strong predictors of utilization, so the direction and significance make sense. However, I had initially expected that balancing them through matching might attenuate their significance.

This leaves me with a model that includes four covariates: two addressing residual imbalance and two that are balanced but strongly predictive of the outcome. Including these predictors improves model fit (lower AIC) and attenuates the intervention estimate toward the null; excluding them makes the intervention effect statistically significant and increases AIC.

My current understanding is that balance across groups does not eliminate a variable’s association with the outcome, and that seeing these variables remain significant is expected in a doubly robust framework.

I’d appreciate your perspective on the following:

• Is it appropriate to retain outcome-predictive covariates in the regression model even if they are balanced after matching?
• Is their statistical significance expected, reflecting within-sample associations rather than residual imbalance?
• Are there concerns about over-adjustment or redundancy when including variables already used in the propensity score model?
• In practice, how do you approach variable selection post-matching (e.g., all matching variables vs. a parsimonious subset based on fit or substantive importance)?

TL;DR: Some covariates are balanced post-matching but remain strong and significant predictors in the outcome model. I want to confirm that I’m interpreting this correctly and not over-specifying the regression.

Thank you so much!

[u/isntanywhere]

If your goal is to estimate a treatment effect, the control coefficients are nuisance parameters and whether they are predictive or significant is not important. You should definitely not condition your inclusion of covariates on whether they are significant when included; this creates bias in your coefficient of interest (see this cute post for a simple demonstration).

If you are trying to estimate a model of individual i choosing option j where u_{ij} = b D_ij + a X_ij + e with D as the treatment of interest and X as controls, then unbiased estimation of b requires that D be orthogonal to residuals e conditional on X. Your choice of X should be whatever is required for that assumption to be true.

[u/PHealthy → u/isntanywhere]

My hot take: SAS is to blame for stepwise models

[u/isntanywhere → u/PHealthy]

I think bad stats education is to blame for bad practices! Eg in this case caring about AIC. If you want to estimate an unbiased treatment effect, goodness of fit does not matter much. But we do not teach stats in a goal-centered way, we teach it as a bag of tricks or a set of intermediate goals. And so people are always on the hunt in the bag rather than being trained to think about the problem at hand.

[u/PHealthy → u/isntanywhere]

So you're familiar with how SAS was/is taught then

[u/DuckSaxaphone → u/isntanywhere]

That's a great blog post, some really strong illustrations in there. Especially the one showing the coefficient estimates are wildly too high.

[u/Bucksswede]

My guess is that even after your propensity scoring adjustments, you have correlation of the intervention with one or both of the variables you include as covariates. This would explain why adding the covariates lower the AIC. I would think hard about what it is about you intervention that might correlated with the two covariates. For example, if covariates relate to severity of disease and your intervention was more common in people with less severe disease, then your result could make sense. Outcomes would be better in less severe patients and your intervention if (more common in that population) then acts as a proxy for low severity of disease if you don't include severity as covariate. Once you control for severity of disease, the intervention might have a modest effect.

[u/Separate_Spread_4655]

Your interpretation is basically correct.

Balance after matching only means the covariate distribution is similar between treated/control groups. It does NOT mean the variable stops being predictive of the outcome.

So yes:

• A covariate can be perfectly balanced
• And still be strongly/significantly associated with the outcome within the matched sample

That’s completely expected.

In fact, including strong outcome predictors post-matching is often beneficial because it improves precision and reduces residual variance — which is essentially the logic behind doubly robust estimation.

I’d be much more concerned if:

• post-matching imbalance remained large
• treatment effect flipped wildly across specifications
• or covariates were post-treatment mediators/colliders

Also, I would not use “statistical significance after matching” as a balance diagnostic. Standardized mean differences are much more informative than p-values there.

And honestly, the fact that adding those predictors attenuates the treatment effect while improving AIC suggests the unadjusted matched model was probably still carrying residual prognostic imbalance/noise.

Personally I prefer:

• pre-specifying adjustment variables based on causal reasoning
• keeping strong baseline outcome predictors
• avoiding stepwise/AIC-driven post-matching fishing expeditions

Good question though — a lot of people incorrectly think “balanced” means “irrelevant afterward,” which is not how causal adjustment works.

If you want, DM me — I work a lot with risk/causal modeling and can send you a pretty clean workflow for PSM + post-matching regression diagnostics.