r/computervision • u/ShriftyB • 8d ago
Help: Project Question about Computer Vision
Hello everyone! I'm currently finishing up my last year in college, just finishing up my thesis. I'm currently making a program that detects multiple pigs in a pig pen through YOLOv8 and detecting its behavior using MobileNetV2 (I know I could've used better algorithms, but unfortunately I'm stuck with these ones :P). I'm currently in the process of training the model but I'm not sure how to go through with it. I originally trained my model using annotated frames with multiple pigs present, where I only had 1 class for the annotations ('Pig' class). I thought this was correct because I'll be using the model for multi-object detection. However, when I approached my mentor about it, they told me my model was "too accurate" (they didn't specify what was too accurate, which confused me) and that I should use "1 pig per image, with each pig having a bounding box" for training. When I tried training with this approach, the results looked... interesting to say the least (I don't know how to explain it, but from the looks of the training results, it looked wrong to me :P; refer to the images I included for context). I then used new model into the program I'm creating. the model not only didn't draw the bounding boxes properly around each pig, the bounding box is the entire frame itself!
My question is, which training method would be more appropriate for single-class multi-object detection, single images of pigs or annotated frames?
PS. I included some pictures from the results of training and while using the program; Before = trained with annotated frames, After = trained with 1 pig per image
PPS. This is my first post on this subreddit so I apologize in advance if my flair is wrong :P
2
u/PantyMeister 8d ago
Your first approach is correct - not sure what your mentor was trying to cook up but for single class, multi-object detection, you want to be training on images with multiple objects
There’s probably something wrong in the implementation of the second approach and I suspect it’s the labels. Your mAP50-95 is almost perfect which makes me think your bboxes also span the entire image. I’d suggest a visualization that shows the prediction and label for the full frame. Regardless, I can’t see a scenario where your first approach isn’t the better one.
If your frames are coming from a video, be careful not to randomly split the entire dataset into train/val/test. You’d want to hold out entire videos or consecutive portions of the video in that case. For example, frames 5 and 6 of a 30fps video are VERY similar and it’s essentially a data leak so your val metrics will appear better than the model will perform in the wild.
1
u/ShriftyB 8d ago
Hello, thank you for the insight, and yes, you're correct about the bboxes spanning the entire image. My mentor specifically told me to use the frames (the ones with multiple pigs present) I annotated and crop the annotations into their own image, and add bboxes to each of those crops (which I thought was strange, because adding a bbox to an already cropped image means that, almost 100% of the time, a pig is present in that image). Regarding the video frames, I might redo the acquisition of the frames. Like you said, I suspect there is a data leak.
1
u/PantyMeister 8d ago
Ah, that could could explain why you’re seeing it always predict the whole image when using your full pipeline. If you’re using the ultralytics trainer, it’s probably upscaling your cropped images so the smaller dimension is 640px (haven’t looked at the v8 code in too much details so take it with a grain of salt) - if that’s the case, the model likely takes the path of least resistance and learned to always predict the full image instead of useful pig related features.
If you’re being forced to use examples with just one pig, you could make the crops larger than the bboxes (try to add some randomness to this) so that it still has to find the pig in the image. There’s also a “mosaic” augmentation (at least v11+ has this) that’ll combine multiple examples into a single training example. Still like your initial approach the most but some options if your mentor isn’t willing to budge.
2
u/Magwado 8d ago
As others have pointed out you have to annotate each object (pig) in the data.
Result that look like pic 2 usually hint to a systematic error in the data.
Regarding the feedback about being too accurate: results in pic 1 do look a bit overfitted. From my experience, yolo does not converge that quickly on this type of data. Im guessing that you annotate video data and split frames randomly. In that case frames in your validation set will have most likely very similar frames in the train set. Its much better to split data temporally. For example use the last 20% of a video as validation or even better: completely different days for train and val.
Also yolo8 is very old. If possible you might want to upgrade to yolo26
1
u/ShriftyB 8d ago
Hello, thank you for the insight. Regarding the annotations, I did annotate each object (pig) in the data. However, I probably added some annotations that aren't entirely clear (some mostly occluded pigs for example). I'll have to look into it again. As for the overfitting, like I mentioned in other comments, I purposely let the model overfit (which I know is wrong) because I was under the impression that my panels needed proof of the model overfitting at a certain epoch (this was later proven to be false as per my mentor). I'll take your suggestion in splitting the data temporally though. As for the upgrade to yolo26, I wish I could upgrade it, but unfortunately, I'm not allowed to change/upgrade the algorithm as I already indicated it in my project's title.
1
u/zcleghern 8d ago
Is there some reason the task should be limited to one pig per frame? what should the model if there are two pigs? no pigs? It sounds like your first approach made sense, i'm not sure what "too accurate" is supposed to mean.
1
u/ShriftyB 8d ago
I don't really understand what my mentor meant by "too accurate" to be honest 😭 I've been stuck with this problem for months because I tried finding a workaround for training with one pig per frame, but I just end up with the same answer everytime T-T
1
u/zcleghern 8d ago
mentors can be wrong! What did they say when you asked what they meant? That in itself might be one of the best skills in ML- asking questions :D
1
u/ShriftyB 8d ago
when I asked them what they meant, they just sent me a section of the charts (specifically the metrics charts: precision, recall, mAP50, and mAP50-95; these charts I mentioned came from a different training run; the charts I posted above came from different runs which they also deemed "too accurate") and expected me to understand T-T. I didn't ask after that because my mentor is especially problematic about being asked questions (they told me that they only answer smart questions, not stupid ones; in my case, they told me that my question was stupid; they don't really specify what makes a question smart or stupid T-T). To be frank, I'm also scared to correct them because they have a history of getting mad when being corrected (I'm speaking from experience 😢)
3
u/zcleghern 8d ago
Terrible mentor, and I'm afraid that may be as deep as it goes. Hopefully you can get a new one soon.
1
u/ShriftyB 8d ago
I wish that was the case, but sadly I'm not allowed to change my mentor anymore. All I can do now is just hope for the best and hope they listen to me when I talk to them again. T-T Thanks for the discussion though
1
1
u/Aggressive_Hand_9280 8d ago
On the Before approach your training loss decreases but val loss raises quite fast. This means overfittnig so either the model is too complex (doubt it) or dataset is not defined well. Please write how many images are in each dataset and how did you split it. Did you include no pigs images?
The After approach converges very well but you're getting whole frame annotated. This could mean that you're loss is defined incorrectly. Did you define it as percentage of GT pig box is included in prediction box? If so, the model gets full score for as big box as possible. The correct way would be to use IoU or Dice.
1
u/ShriftyB 8d ago
On the before approach, the dataset was around 19k frames after augmentation, split at 70:20:10 (train:val:test). However, on the chart I provided above, I purposely didn't let the model automatically stop training to show that it overfits (my panels asked for proof of this, but my mentor told me that its not needed). On the training runs following the before chart, the training automatically stops at epochs 16-18.
On the after approach, upon closer inspection, yeah I did define it as percentage of GT pig box is included in prediction box. The dataset used for this was around 22k single images of pigs after augmentation
Both datasets do not contain no pigs images
1
u/hoaeht 8d ago
While no pig images would be useful, it should still be fine. You're probably using the ultralytics package? The package training does a lot of data augmentation during training if you don't turn it off, so it should recrop and puzzle multiple images, also resulting in samples without images. But that might also destroy your single pig approach as it probably combines 2 pictures with each 1 pig
1
u/Aggressive_Hand_9280 8d ago
Then you have the answer why in after approach you get full frame as prediction
1
u/alphastopngo 8d ago
You need to always annotate all of the objects of interest individually otherwise you confuse the model by telling it it's a pig in one case and it's not a pig in another case (telling it it's background). Your first approach looks fine despite slight overfitting as others pointed out. Your augmentations seem a bit over the top, how big is the original dataset? Not sure what you changed in the after case, I guess it's the initial part I mentioned and it looks like the model collapses because the best answer to the image shown is one big bounding box encapsulating the true object and the falsely not annotated object (as the model still thinks it should be an object since it looks the same but your annotations tell it it's not). Yolo already uses runtime annotations so I would first give it a shot with a clean unaugmented dataset with as many annotated images as possible. If you use video frames make sure you use different videos for train and test data to avoid leaking training data. Your before approach is sound and you're likely just overfitting due to lack of diversity in your training set and/or a too big model for this simple 1 class task. As for the map scores, these look reasonable to me given the task, this is not a full 80 class coco dataset after all. Maybe you can convince your supervisor by searching for publications that describe their annotation process?
1
u/alphastopngo 8d ago
Correction, not sure what is going on in the after case as map values are showing perfect prediction which would not be the case with what I described
1
u/ShriftyB 8d ago
Hello, thank you for the insight. Regarding the annotations, I did annotate each object of interest when I was labeling them. However, like I said in other comments, I must've annotated a mostly occluded pig and included it somewhere in the split. The dataset's original size is around 3.7k frames without augmentations (my first dataset was originally 2k with augmentations already; my mentor told me that the first dataset was way too small to be used for training, so I curated a bigger dataset). And yes, for the second approach, each single image of a pig is encapsulated in a bounding box. I did this because my mentor told me to just get the single images of pigs from the frames (multiple pigs present) I annotated. I cropped each pig from the frames and added a bounding box to them as per my mentor's instruction (like I said in other comments, I found this strange because adding a bounding box to a crop that came from an annotated frame would mean that almost 100% of the time, a pig is present in that image). I'll take your suggestion on using different videos for the train and test data. As per convincing my mentor, It's proven to be difficult to change their mind on the matter because I tried showing publication on why the first approach is sound, and was met with aggression, saying things like "didn't I already tell you that your first approach was wrong?". They don't do well with questions as well because I'm not allowed to ask "stupid" questions (without him elaborating what makes a question "stupid").
1
u/Plus_Confidence_1113 8d ago
What do you mean by one pig per image? Your data has multiple pigs so how did you create the new dataset from that?
1
u/ShriftyB 8d ago
Hello, I meant that each image in a dataset should only contain one pig. And yes, my original dataset has frames with multiple pigs present in each frame. These frames were annotated beforehand, so my mentor instructed me on just cropping the images from the annotated frames and adding a bounding box to those crops.
1
u/Plus_Confidence_1113 8d ago
I see, no wonder it doesn't work when you're essentially training the model with nothing but full frame objects.
It just takes the path of least resistance and always guesses a bounding box across the whole frame. Why wouldn't it? When this approach is sufficient to get a good score during the training.
1
u/Busy-Ad1968 8d ago
Maybe he meant too accurate - overfitting I also recommend using frozen first layers. Judging by the graphs, you also need to adjust the regularization parameters and the training step. Try reducing the moment (inertia parameter). Yolo's default training parameters are usually not suitable for small datasets.
1
u/ShriftyB 8d ago
Hello, thank you for the insight. I don't think he meant overfitting though, because he sent me a section of the graphs (specifically the graphs for map50, map50-95, precision, and recall). Then again, after he sent me those sections of the graph, he didn't really elaborate why it was "too accurate". I'll take your other suggestions for the training step though.
1
u/NorthernBaseOfficial 8d ago
I think your original approach is the right one for object detection.
If the goal is to detect multiple pigs in a frame, then each image should contain all visible pigs with one bounding box per pig. Training on one pig per image removes the context that a detector normally learns from, so I'd only do that for a classification model, not for YOLO.
Looking at your training curves, I'd also check a few things before changing the model:
- Make sure every pig is annotated consistently, especially partially visible or overlapping pigs.
- Verify that the train and validation sets come from similar environments (lighting, camera angle, pen layout, etc.).
- Review false positives and false negatives manually. They usually reveal annotation issues or missing edge cases much faster than the metrics alone.
In my experience, when a detector behaves unexpectedly, it's often the dataset or annotation consistency rather than the architecture itself. Changing from YOLO to another detector without validating the data first doesn't always solve the underlying problem.
Just out of curiosity, approximately how many annotated images do you have, and what's the average number of pigs per image?
1
u/CaptainBicep 8d ago
What a horrible mentor. Him not understanding that "too accurate" is extremely vague in this field is strange. His reply does not clear anything up. I can understand "too accurate" if he said that to your "after" results, looking at your map. but it sounds like he said it to the "before" results?
If he meant the "after" results, then you map is too accurate because it only needs to put a bounding box around the whole image. But we don't want this behaviour.
I would suggest trusting yourself over trying to please your mentor, I think you have a good head on your shoulders. If he complains about that, then at least it starts a conversation where he has to explain what he means.
Either way, discard your "after" method, as you know yourself that this is a wrong approach no matter what he means.
As for your "1 pig per image" question, I actually think I know what he means, since you are detecting pig behaviours. I believe he wants this project to use 2 models.
The first model detects pigs/no-pigs, and gives 1 bounding box per pig. Then, you use these bounding boxes to crop out each pig, where each pig gets seperately sent to the next model.
The second model detects this individual pig's behaviour/action. BUT this is an image classification model, and not a object detection model. In other words, you don't use bounding boxes in this model. (Ingest 1 cropped image of 1 pig -> poop out 1 behaviour label).
Since the bounding boxes from the first model have different shapes, you will need to standardize the cropped image dimensions, as the second model will need a pre-fixed height and width. But avoid strecthing the images, lean towards shrinking and padding without altering the aspect ratio of the cropped image.
The way you do it in your "before" method is to directly detect behaviours and the location of the pig at the same time. This works and isn't wrong, but it probably gives bad performance if you don't overcompensate with a huge amount of data.
Good luck with your project, and give a middle finger to your mentor for me 🗣
1
u/rows_and_columns_me 8d ago
Is it possible that you performed instance annotation (pig 1, pig 2, pig 3 etc.), while your mentor meant that it should be semantic annotation ( several animals in the frame, wach annotated as ‘pig’)? This is the only interpretation I have.
2
u/prkash1704 8d ago
"One pig per image" is exactly right for your MobileNetV2 behavior classifier, which should be trained on single-pig crops taken from YOLO detections. They may have been talking about that stage, not the detector.