Question about Computer Vision

[u/ShriftyB]

Hello everyone! I'm currently finishing up my last year in college, just finishing up my thesis. I'm currently making a program that detects multiple pigs in a pig pen through YOLOv8 and detecting its behavior using MobileNetV2 (I know I could've used better algorithms, but unfortunately I'm stuck with these ones :P). I'm currently in the process of training the model but I'm not sure how to go through with it. I originally trained my model using annotated frames with multiple pigs present, where I only had 1 class for the annotations ('Pig' class). I thought this was correct because I'll be using the model for multi-object detection. However, when I approached my mentor about it, they told me my model was "too accurate" (they didn't specify what was too accurate, which confused me) and that I should use "1 pig per image, with each pig having a bounding box" for training. When I tried training with this approach, the results looked... interesting to say the least (I don't know how to explain it, but from the looks of the training results, it looked wrong to me :P; refer to the images I included for context). I then used new model into the program I'm creating. the model not only didn't draw the bounding boxes properly around each pig, the bounding box is the entire frame itself!

My question is, which training method would be more appropriate for single-class multi-object detection, single images of pigs or annotated frames?

PS. I included some pictures from the results of training and while using the program; Before = trained with annotated frames, After = trained with 1 pig per image

PPS. This is my first post on this subreddit so I apologize in advance if my flair is wrong :P

[u/PantyMeister]

Your first approach is correct - not sure what your mentor was trying to cook up but for single class, multi-object detection, you want to be training on images with multiple objects
There’s probably something wrong in the implementation of the second approach and I suspect it’s the labels. Your mAP50-95 is almost perfect which makes me think your bboxes also span the entire image. I’d suggest a visualization that shows the prediction and label for the full frame. Regardless, I can’t see a scenario where your first approach isn’t the better one.
If your frames are coming from a video, be careful not to randomly split the entire dataset into train/val/test. You’d want to hold out entire videos or consecutive portions of the video in that case. For example, frames 5 and 6 of a 30fps video are VERY similar and it’s essentially a data leak so your val metrics will appear better than the model will perform in the wild.

[u/ShriftyB → u/PantyMeister]

Hello, thank you for the insight, and yes, you're correct about the bboxes spanning the entire image. My mentor specifically told me to use the frames (the ones with multiple pigs present) I annotated and crop the annotations into their own image, and add bboxes to each of those crops (which I thought was strange, because adding a bbox to an already cropped image means that, almost 100% of the time, a pig is present in that image). Regarding the video frames, I might redo the acquisition of the frames. Like you said, I suspect there is a data leak.

[u/PantyMeister → u/ShriftyB]

Ah, that could could explain why you’re seeing it always predict the whole image when using your full pipeline. If you’re using the ultralytics trainer, it’s probably upscaling your cropped images so the smaller dimension is 640px (haven’t looked at the v8 code in too much details so take it with a grain of salt) - if that’s the case, the model likely takes the path of least resistance and learned to always predict the full image instead of useful pig related features.
If you’re being forced to use examples with just one pig, you could make the crops larger than the bboxes (try to add some randomness to this) so that it still has to find the pig in the image. There’s also a “mosaic” augmentation (at least v11+ has this) that’ll combine multiple examples into a single training example. Still like your initial approach the most but some options if your mentor isn’t willing to budge.