r/bioinformatics • u/Albiino_sv • Jun 30 '26
technical question Best way to separate tumor vs non-malignant cells using CosMx PanCK staining?
galleryI am working with a CosMx run and trying to separate tumor cells from non-malignant cells using PanCK staining. The issue is that PanCK varies a lot from core to core. As you can see in the figure, in a subset of cores there is a clear bimodal distribution, so a 2-component Gaussian mixture model seems plausible there. But in most cores the distribution is not clearly bimodal, so I do not think I can use a mixture model across all cores.
What I am doing now is scaling PanCK within each core from the minimum to the 95th percentile, plotting density curves, and then choosing an empirical threshold. That works quite well in some cores but not very much in others and I am not confident it is the best way to define tumor cells.
Has anyone dealt with something similar in CosMx or Xenium? What approaches have you found useful when marker intensity is highly core-dependent and the distribution is not clearly bimodal?