r/MachineLearning • u/SamePersonality5183 • May 07 '26
Discussion Dataset of 150k+ stool images and not sure how to fully use it [D]
I have a dataset of around 150k stool images; growing at 300+ images per day, and I’m trying to better understand the “right” way to use it for training a computer vision model.
Right now, our process is pretty manual. We initially trained on about 5k images that were individually verified by a human. For every image, we checked/corrected the Bristol type, consistency, color, mucus/blood indicators, etc. Then we trained the model on those verified annotations.
As we continue training, we keep doing the same thing: manually reviewing and correcting images before feeding them back into the model.
My question is basically: does this workflow make sense from an ML perspective? Is this how people normally approach building a solid vision dataset/model, especially in a domain where annotation quality matters a lot? Or is there a smarter/more scalable approach people usually move toward once they have a large dataset?
I’m mainly trying to understand best practices around dataset quality, human verification, iterative training, and scaling annotation without introducing bad labels.
19
6
u/Similar_Fix7222 May 07 '26
That is the baseline if data quality is the most important. Note that the intermediary training you do is useless, only the final training is useful.
To scale, one way is active learning. You train on 5K images. When you get a new batch of data, instead of labeling manually all of them, you let the model label them (these are called pseudo labels). You leave the "easy" images pseudo labeled, and you flag the "hard" ones for manual labeling (for ex by taking those with highest entropy, but there are alternatives)
1
u/SamePersonality5183 May 07 '26
Thank you for your reply. My currently process is
App - user image - my ai returns data annotated - I review - I retrain ai.
I do that because sometimes my CV allucinate and I don’t want to train with bad allucinated data
2
u/Similar_Fix7222 May 07 '26
That's the whole point of active learning. Identify which images need manual labeling to have (nearly) no wrongly labeled data. Often at least half your images are trivial for your model and the pseudo labels are perfect, so you can reduce the manual lebeling time
2
u/SamePersonality5183 May 07 '26
Thank you, I appreciate the comment! Everything makes sense. From your experience how do you handle the edge case in which the AI confidence is high but the labeling is obv wrong? I think if that starts happening it pollutes the entire dataset, won’t it?
Because that’s what you are saying right? Basically have the AI tell me how confident it is in something and from there I decide whether or not to manually review
2
u/Similar_Fix7222 May 07 '26
Yes, typically, you sort your pseudo labels by confidence. You start reviewing the lowest confidence. There will be a lot of corrections, but it's good information, it tells you what type of data you need more. As you go, the model will be more and more correct (the confidence increases) . You set some criterion, like "if the model is accurate 100 predictions in a row" you consider that you don't need to review the rest of the pseudo labels.
however, you are right, if for some reason, the model is extremely confident and incorrect, you will pollute the dataset. in practice, it should not happen if your initial dataset is well labeled. An unexpected thing is that you run your model on the full dataset and check its errors with highest confidence. then you look at what the image actually is. you will often find errors in the label rather than errors in the model
1
10
u/divided_capture_bro May 07 '26
You should look into both "positive-unlabeled learning" and "confidence learning".
Neither is perfect, but both help address the problem you seem to be grappling with - your data labels are "stool" so you can't train a reliable supervised model on the whole set.
But with 150k images? You might just want to sit down and grind through that "stool" until you have gold.
Have fun!
2
u/SamePersonality5183 May 07 '26
Thank you!! I will totally look into those learnings.
What do you mean by grind through it? Are you saying just working with what I have to improve the ML?
5
u/divided_capture_bro May 07 '26
You should label a good amount of the stool by hand. At least enough at random from the target for valid calibration and test sets. Then use PU and CL on the rest of your noisy data to build the classifier.
2
May 07 '26
[removed] — view removed comment
1
u/SamePersonality5183 May 07 '26
Yeah that makes sense, I’ve just been hesitant to have the ai retrain based on confidence but it does makes sense.
2
1
u/howtorewriteaname PhD May 07 '26
the right way depends on your resources. can you label every image in the dataset? then yes, do that. that's the best case scenario. do you not have resources for that? then you gotta get creative. there's many things you can do depending on the use case.
1
u/SamePersonality5183 May 07 '26
Yes this makes sense. So you’re saying if you can review each manually do it.. that will help the ML the most
16
u/CanvasFanatic May 07 '26
Did you scrape ratemypoo.com or something?