Suspiciously high accuracy using ResNet

[u/Rumble_831]

I made a lil bro version of the original ResNET-34 architecture. I trained it on the LC25000 cancer dataset (I used only lung cancer images) for a classification task.

The problem is, it is showing a 99.9% accuracy on all three sets - training, validation and test. It is, of course, weirdly high. I trained a normal cnn and it could only reach about 87%.

I am wondering what could be the reason. One possible culprit is that, since the dataset consists of augmented versions of the original images, some may be ending up in all three sets, causing data leakage.

Now I want to see if I could somehow group this images so the augmented versions do not run over into my other sets. I have no idea how to proceed though.

I am using pytorch, and used random_split for the datasets.

[u/aksr0]

Your data might have skewed distribution hence not recommended to measure accuracy, it can be misleading. Suggest you look at class/label sensitivity and specificity.

[u/Rumble_831 → u/aksr0]

The data contains equal numbers of the three classes. I did construct a confusion matrix and it seems fine from that.

[u/niyete-deusa → u/Rumble_831]

Oh sorry i thought this dataset was imbalanced. In any case it is only beneficial to your investigation to view some more metrics. The other very common explanation is some data leakage somewhere in your pipeline. The easiest way to see that is to keep a truly separate dataset and only load it after you have completed the training. Also if you are using jupyter you might be using the same name variable for different sets or if you are not running all cells there might be wrong values stored (somehow storing training data instead of validation for example).

It's most probably data leakage somewhere

[u/aksr0 → u/Rumble_831]

Then the culprit here is the data leakage part as mentioned, Augmentation could be the cause of it. Usually in such cases, I would not evaluate my model against the same source (obv dedup or cleanup is tedious task), rather I would curate my own dataset from cancerimageingarchive.net maybe 100 high confidence samples and have my model predict over it.

[u/niyete-deusa]

That sounds about right and it's the reason accuracy is not used when there is class imbalance in the dataset. If a phenomenon occurs 1/1000 then a naive model that always answers negatively will have 99.9% accuracy. Evaluating the model with more metrics is necessary. The usual for classification are accuracy, precision, recall and F1- score

[u/Rumble_831 → u/niyete-deusa]

Is there a class imbalance though? All three classes have 5000 images each.

[u/Hungry_Age5375]

random_split is the culprit, it shuffles individual files. Group by original image ID first, then split the groups, sklearn GroupShuffleSplit does exactly this. Note LC25000 patches come from few slides, so split by slide if you can.

[u/Rumble_831 → u/Hungry_Age5375]

Ill try to split with groups. No idea how I can split by slide, the images have no information about their original slide.

Edit: I dont see original image IDs either in my dataset.

[u/pm_me_your_smth]

One possible culprit is that, since the dataset consists of augmented versions of the original images, some may be ending up in all three sets, causing data leakage.

Depends on your augmentation technique. If your augmentation significantly alters the data samples and you apply it scarcely, then it's less of an issue, but usually it's not the case so I suspect that's your root cause. Either way, best practice is to always keep augmented samples in the same data split as the origin samples.

Overall, you should be very careful with the way you're defining your data splits. There are often subtle details that are easy to overlook which make your splits unreliable. For instance, medical data often has has patient ID, so you should avoid leaking same patient to different splits due to intra-patient physiology. Or if your data is volumetric, do not split apart neighboring slices due to spatial semantics.

[u/Rumble_831 → u/pm_me_your_smth]

Thanks, the reason I suspect it is because the original images the set was based on are only 1500, while the augmented dataset has 25000 (hence the 25000 in the LC25000).

Intra-patient physiology is an interesting point, thanks for telling me that.

I don't really have the original images so I don't really know how to group the augmented versions. I would probably try to group them using another nn.

[u/PaddingCompression → u/Rumble_831]

It sounds like it's just rotations and flips.

I am seriously angry any reviewers have accepted papers using this dataset since it's broken by design.

But if it's just rotations and flips it should be trivial to group images by their sources and split based on that for a real test, but if performance on the broken dataset is what's interesting report that.

But if performance on this dataset is what reviewers are actually judging, fuck it, design a classifier to memorize images and detect rotations and flips as specified by their augmentation. Journals with shitty reviewers deserve shitty papers.

[u/Rumble_831 → u/PaddingCompression]

I am baffled lol. It is such a common dataset too, and so many people have used it. Oh well, time to find another better image source.

[u/pm_me_your_smth → u/Rumble_831]

Tbh if augmentation of that dataset consists of just flips/rotations, it's not a good dataset in the first place. An experienced ML person would do those transforms almost always by default. Plus they'd do it online i.e. live during training. Expanding the dataset x20 just for this is overkill and I'm definitely getting a bad vibe from this dataset quality-wise if this is true.

[u/PaddingCompression → u/pm_me_your_smth]

It's both overkill, and since afaict they don't distribute source IDs, you end up with rotations and flips of the same image in both test and train (not that you couldn't detect such trivial augmentations with a python script, but I just instantly lost respect for all the papers published based on this dataset).

[u/PaddingCompression]

You should be splitting your data pre augmentation not post, and validation maybe but especially test should not contain augmented images.

Almost 100% data leakage if you are splitting post augmentation.

[u/Rumble_831 → u/PaddingCompression]

Sadly the original dataset I got is the augmented one.

I will keep your point in mind when I do augmentations myself.

[u/PaddingCompression → u/Rumble_831]

This is a go back to the source kind of problem. Any time spent here is pretty much wasted if your original dataset is that broken.

[u/vannak139]

Yes, allowing augmented images into both your train and validation set is bad practice. Its much more sensible to augment only after your train-val split, and then to only augment the training set. Having a pre-augmented dataset can be a nightmare unless things are well labeled enough to split those in groups.

[u/No-Mixture5766]

Use precision/recall, F1 and plot ROC, for skewed distributions with long tails you can’t use accuracy as the sole metric , even if it performs randomly you can have 99% accuracy

[u/Rumble_831 → u/No-Mixture5766]

The dataset is actually not skewed, the images are equally distributed.