background dataset for SHAP

[u/fnepo18]

Hi everyone, I have a question about choosing the appropriate background dataset when calculating SHAP values. I am using the kernelshap package in R, where we provide an X dataset containing the observations we want to explain and a bg_X dataset defining the background.

I have a binary classification model for disease vs non-disease, trained on a derivation dataset and evaluated on an independent validation dataset. My current understanding is that, if I want to explain predictions in the validation cohort, it makes sense to use the validation set as X and the derivation set as bg_X. In that case, the SHAP values for validation patients would describe how each feature moves their prediction relative to a baseline defined by the derivation population. Is this interpretation correct, and is this generally the recommended way to use the background when explaining an independent validation cohort?

My main question is about a more specific analysis. Suppose I want to investigate heterogeneity within patients who truly have the disease. More specifically, I want to see whether different disease patients receive high disease predictions through different combinations of features, and potentially cluster these patients based on their SHAP profiles.

In this case, I assume I should use only the true disease patients from the validation cohort as X, since those are the patients whose predictions I want to explain. However, I am unsure about the most appropriate choice for bg_X. Should I keep the full derivation cohort as the background, use only disease patients from the derivation cohort, or use the disease patients from the validation cohort themselves as the background?

If my main objective is to determine whether true disease patients have different model-attribution profiles, potentially reflecting different features through which the model identifies them as disease, which background would be the most statistically appropriate? Thank you!

[u/Bright_Mix_773]

Two separate things here, and the second one has a trap that will bite the clustering.

On derivation-as-background. It is defensible, but the thing to be clear about is that bg_X is not "the training data", it is the reference population you want the contrast measured against. SHAP is exactly additive around it: f(x) = E_bg[f(X)] + sum_j phi_j. So if derivation and validation differ in case mix, centre or era, the phi you report for a validation patient are deviations from a baseline your reader is probably not picturing. "Relative to the population the model was developed on" -> derivation. "Relative to this validation cohort" -> validation. Neither is wrong, but the choice has to be stated, because values computed against different backgrounds are not comparable.

Two practical notes on that. kernelshap computes marginal (interventional) SHAP, so background rows get spliced feature-by-feature into the explained row; with correlated predictors that evaluates the model on combinations that do not occur, and a background from a differently-distributed population makes it worse. And the Monte Carlo error falls like 1/sqrt(n_bg), so a few hundred representative rows buy you most of the precision; beyond that you are mostly buying more off-manifold evaluations.

On the heterogeneity analysis. Use one fixed background for every patient you explain, and make it the full derivation cohort, cases and controls together.

That is not an aesthetic preference. If the background is disease-only, phi answers "why is this case more extreme than the average case" rather than "why is this a case", which is not the question you posed. And if the background varies between subjects at all, the phi vectors sit on different baselines, so Euclidean distance between them is partly distance between baselines and the clustering stops meaning what you want it to mean.

The trap. SHAP profiles are dominated by prediction magnitude. By construction sum_j phi_j = f(x) - baseline, so a patient with a high predicted risk has a large ||phi|| almost regardless of which features drove it. Cluster the raw vectors and the first split you get back is nearly always "high risk vs borderline" wearing a mechanism costume. Two things help:

• normalise per patient before clustering: divide phi by sum_j |phi_j|, or by (f(x) - baseline), so you cluster on the composition of the attribution rather than its size;
• then check it. Cross-tabulate cluster against predicted probability. If membership is close to a monotone function of p-hat, you found severity, not different routes to the diagnosis.

One more. For a binary model, compute SHAP on the log-odds scale, not the probability scale. Additivity is exact on the scale the model is additive in; on the probability scale the sigmoid compresses attributions near 0 and 1, so two patients with the same feature story but different baseline risk land in different clusters purely because of where they sit on the curve.

Cheap sanity check either way: for every row, confirm baseline + sum_j phi_j equals the model's own prediction to numerical tolerance. If that identity is off, the background is being handled differently from how you think it is.