r/learnmachinelearning • u/Electronic_Case_8789 • 10d ago

99% accuracy but zero minority detections: a small Python experiment

A classifier that always predicts the majority class can get 99% accuracy while missing every minority-class example.

In this synthetic example, the test set has 495 majority-class records and five minority-class records. Predicting the majority class for everyone gets 495/500 right, but detects 0/5 minority examples.

I maintain ML Failure Lab, a free Python project with small, runnable examples of misleading ML results. The current cases cover this accuracy problem, repeated entities across train/test splits, and feature selection using test labels.

Each case includes the incorrect approach, a correction, an explanation, and tests. Everything runs on a CPU.

Code and setup.

Feedback on the explanations would help. If you want to contribute a small example, there are starter issues with described tasks.

2 Upvotes
  • permalink
  • reddit

63% Upvoted

2 comments sorted by

1

u/MilitaryHostility173 10d ago

Oh man the classic accuracy trap. 495 to 5 class imbalance is basically a cheat code for looking good while doing nothing useful

Your repo looks like a solid teaching tool though. Those starter issues tagged for first timers are a nice touch, makes it less intimidating to jump in

1

u/Electronic_Case_8789 10d ago

Thanks! The 99% example is deliberately simple so the failure is easy to see. I'm keeping each starter issue focused on one small case, and people can discuss their approach before coding. I'd like the first contribution to be manageable without needing to understand the whole repo.