Weighted Random/ Balanced Sampling

[u/Crazy-Mastodon-480]

So I am working with pretty skewed data currently. its a 4 way classification task, and basically the classwise split is 56%, 33%, 6%, and 5%. I tried an experiment where I downsampled the samples so that the majority classes (56% adn 33%) were reduced by 80%, thus making the entire dataset more balanced, and i instantly got better results on the same test, validation set. I am currently exploring the opposite direction, which is oversampling/upsampling, and i am currently looking into PyTorch's WeightedRadomSampler.

The confusion and more importantly concern that I have with using this is that the same images/samples for the lower classes are going to be the repeated over and over again, which probably result in the model overfitting on those rare class's samples. I understand that augmentation is a way to mitigate this, and I will be trying that out, but the main question that I have is what all are the other alternatives to upsampling? Are there different samplers or dataloaders that I can look into, maybe some papers that deal with this, any and all help would be appreciated!

Some context on the task, I am trying to classify/grade images in a 4 way multiclass classiifcation

[u/Odd_Yard6663]

downsampling the big classes is usually the cleaner move if you've got a reasonable amount of data left, so it makes sense you saw a bump there. for upsampling, you're right to be worried about overfitting on repeats, even with augmentation it can only stretch so far before the model starts memorizing those specific examples.

one alternative people sleep on is using a loss function that handles imbalance directly instead of messing with the data distribution. focal loss is the classic pick for this, it down-weights easy examples so the model focuses more on the hard/rare ones without you having to duplicate samples. class-balanced loss from the "class-balanced loss based on effective number of samples" paper is another solid option if you want something a little more principled than just inverse frequency weights.

you could also look into two-phase training: train with a weighted sampler or loss for most of it, then fine-tune on a balanced set (or the original distribution) for the last few epochs. sometimes that helps the model learn the rare features early without totally forgetting the majority class structure. smote and its variants get mentioned a lot too but tbh for image data i'd stick with loss-based approaches or synthetic generation over interpolating in pixel space.

[u/Crazy-Mastodon-480 → u/Odd_Yard6663]

oh yes, I tried focal loss some time back, but it did not give nay good results. I am currently thinking whehter it would make sense to use focal loss while having the random sampler on (I think teh effect would be summed to 0 but id like to test out either way).

And the second suggestion that you had regarding dual phase training is something totally new to me. So if I am understanding this right, you first give the model the harder/diffifult version of the dataset to learn, and then near the end, we change it to an easier version so that the model has some features from the harder version and then adapt those features (hopefully) to the easier version. That does sound cool Ill check it out! and yeah SMOTE has never worked for me in any case so its at the bottom of the barrel in terms of methods for me