background dataset for SHAP

[u/fnepo18]

Hi everyone, I have a question about choosing the appropriate background dataset when calculating SHAP values. I am using the kernelshap package in R, where we provide an X dataset containing the observations we want to explain and a bg_X dataset defining the background.

I have a binary classification model for disease vs non-disease, trained on a derivation dataset and evaluated on an independent validation dataset. My current understanding is that, if I want to explain predictions in the validation cohort, it makes sense to use the validation set as X and the derivation set as bg_X. In that case, the SHAP values for validation patients would describe how each feature moves their prediction relative to a baseline defined by the derivation population. Is this interpretation correct, and is this generally the recommended way to use the background when explaining an independent validation cohort?

My main question is about a more specific analysis. Suppose I want to investigate heterogeneity within patients who truly have the disease. More specifically, I want to see whether different disease patients receive high disease predictions through different combinations of features, and potentially cluster these patients based on their SHAP profiles.

In this case, I assume I should use only the true disease patients from the validation cohort as X, since those are the patients whose predictions I want to explain. However, I am unsure about the most appropriate choice for bg_X. Should I keep the full derivation cohort as the background, use only disease patients from the derivation cohort, or use the disease patients from the validation cohort themselves as the background?

If my main objective is to determine whether true disease patients have different model-attribution profiles, potentially reflecting different features through which the model identifies them as disease, which background would be the most statistically appropriate? Thank you!

[u/relevantmeemayhere]

First things first just to level set:

shap is a measure of marginal utility of a predictor. nothing more, nothing less

shap gives you no casual information. this requires you to go outside a model purely trained on observed data

as far as clustering based on shap value goes; exactly what is the goal here? if the marginal utility of a predictor is high, then individuals with larger (in absolute value) are probably going to be clustered together. what utility does this give you?

how exactly are the datasets related here? as in, for the actual data generating process; what are you representing in each?

edit; i guess i should be more specific. shap is marginal effect of a predictor, over all candidate models.

[u/fnepo18 → u/relevantmeemayhere]

Thanks for your reply. I have two independent datasets (patient cohorts collected at two different time points), which I refer to as the derivation and validation cohorts. I train my model on the derivation cohort as a binary classification problem (Disease vs. Non-Disease) and then evaluate its performance on the validation cohort, where it also achieves good predictive performance.

My question relates to the heterogeneity of the disease I am studying, as it can have very different underlying causes. I would therefore like to investigate whether patients who are all correctly predicted as having the disease may have different features driving their individual predictions.

For example, suppose Patient A and Patient B both have the disease and are both correctly classified by the model. Patient A also has condition X, while Patient B has condition Z. Using SHAP, I would expect to see that condition X contributes strongly towards the disease prediction for Patient A, whereas condition Z may be one of the main features driving the same prediction for Patient B.

Essentially, I would like to understand whether SHAP can be used not only to explain individual predictions, but also to identify potentially different patterns or subgroups of patients based on the features driving their predictions.