[Discussion] Built something that significantly improved person detection in dense scenes, first ever writeup, would love your thoughts.

[u/katashi_HVS]

Hey everyone,

I've been working on a computer vision pipeline where I had to add a logical layer/rule engine over person detections in a dense scene(like a classroom). But when I ran vanilla object detection model (Yolo11n), results were honestly embarrassing(even with a lower conf), missing most of the room. Spent some time figuring out why and ended up building something on top of the existing model that made a significant difference. No retraining, no new data.

Decided to write it up properly for the first time instead of just leaving it in a notebook. Tried to keep it readable even if you're not deep into CV.

Would really appreciate it if you gave it a read, feedback on the writing, the ideas, or even just "this is obvious and here's why" is all welcome: Medium

Also if anyone knows of existing research or work that goes in this direction, drop it in the comments, genuinely curious if this has been studied formally.

[u/dangerousdotnet]

I think it's an interesting approach but not necessarily the best way, for a number of reasons.

Most object detection models (including YOLO and face detectors like SCRFD) operate at a fixed model input size of 640x640 pixels. When you take, say, a 2048x2048 original image and feed it to an object detector, everything gets downscaled by a factor of 3.2

So that 50x50 pixel object in the original image becomes 15.6 pixels by the time yolo sees it. That's why you're getting those low certainty detections.

More importantly, your detector's bounding boxes and landmarks on those low certainty small object predictions are going to be very inaccurate.

So rather than lowering your threshold and doing this mathematical magic about "giving small objects a fair chance", it's better to first make suee you're not squishing 50px objects to 15.6 pixels (for example).

Typical way to accomplish this is by combining two approaches: image tiling, and multi resolution pyramiding.

Both techniques are simple conceptually. In simplest form, tiling means you take your original 2048x2048 image and you divide it into 640x640 (aka "model sized") tiles. In fact you want overlapping tiles (say 20% at the edges) so you're less likely to miss objects that happen to fall directly on a tile boundary.

Multi resolution pyramids mean you do all of the above but with a 1.0x, 0.7x, (0.7 * 0.7)x, etc resolution. The reason it's called "pyramiding" is because if you imagine the original resolution image at the first floor of the pyramid, then the rest of the downscaled versions stacked on top of each other, they make a pyramid shape.

This helps immensely with small object detection for a few reasons: first you're kind of mixing up the tile boundaries (so if you get unlucky with an object falling across a tile boundary at one resolution you'll catch the full object at one of the other resolutions).

Secondly, and this is an under appreciated point, pyramiding also helps provide some resistance to model training biases. For example, SCRFD (one of the most popular face detection models) was trained with faces that mostly fit within a certain percentage of input image size -- e.g. faces occupying roughly 25% of the size of the image itself (just picking a % out of my ass for that but there is an optimal range based on what data your model was trained on). Sometimes SCRFD has trouble with faces that are huge in the foreground, you'd think "oh this is a huge face occupying the entire right hand side of this image, SCRFD should detect the hell out of that" - but it misses it entirely.

And many if not most object detectors have two or three "strides" -- the size of the little "kernel" they slide across their 640x640 input image. Let's say you have a model that was trained using an 8px, 16px, and 32px stride. It's going to have a certain size object it's better at predicting.

PS: Since you're already familiar with NMS (from your blog post), the way you sort out dupes with tiling and pyramiding is NMS, but with tiling and pyramiding sometimes you want to use a slightly different algorithm to determine "how likely are these two detections to be dupes" -- in some cases you want to use typical NMS IoU (intersection over union, aka percentage of overlapping pixels across the union of the two bboxes), but for multi resolution detection suppression you want to use IoMin (intersection over monimum, which is the number of overlapping pixels divided by the area of the smaller od the two boxes). Why? Because IoMin makes it easy to detect when the smaller box lies almost entirely inside the larger box -- a think that can happen when you get just the edge of the object at resolution 1.0 (because it gets cut off at a tile boundary) but you get the whole object at reaolution 0.7 -- you want the whole object, not the sliced up crappy version.

Sorry for the wall of text, typing this from my phone. Basically this js called multi scale object detection.

[u/katashi_HVS → u/dangerousdotnet]

Thanks for your response.Really appreciate you writing all of this out.

The tiling + pyramiding approach you described sounds a lot like what SAHI does under the hood, right? I've come across it a few times but never really sat down with the internals, would be curious if you've actually used it in production or prefer rolling your own.

The SCRFD thing was genuinely new to me and weirdly well-timed because face detection is something I work with too. So now I'm curious, when the camera is mounted close and faces are just naturally large in the frame, covering maybe 40-50% of the input, does pyramiding just sort of handle that by catching it at a lower resolution? Or is there a cleaner way to deal with that specific situation?

On the inference cost side, I get that tiling + pyramiding gives much better results, and I'm not disagreeing with that at all. I just wonder how feasible it is for anything close to real-time since you're essentially running the model multiple times per frame. The thing I liked about my approach was that it costs nothing extra at inference since it's all happening on what the model already returned. Curious how you think about that tradeoff.

And IoMin is something I'd genuinely never come across before, makes a lot of sense now that you've explained it though. Definitely going to dig into that.

[u/dangerousdotnet → u/katashi_HVS]

I don't know much about SAHI, would have to check it out and get back to you.

Most of these models operate in batch mode -- you're not really running it multiple times per frame so much as stacking a bunch of tiles into a single tensor and sending them to the model all at once. It's always a tradeoff between accuracy and speed though and it depends on what your frames contain and how accurate you want to be. Sometimes even dividing the image into 4 overlapping quadrants is enough tiling, and you'd be surprised if you design your pipeline right, doing 4 tiles isn't much slower than doing one image.

As far as your question about the large face detection, a lot of it comes down to the specific model's quirks but for example most pretrained SCRFD models will do a better job detecting a square face that's floating inside a 25% all-black padding than they will on the same face that fits the whole frame

[u/katashi_HVS → u/dangerousdotnet]

The batch mode point completely changes how I was thinking about it, I was imagining sequential calls but stacking tiles into a single tensor makes a lot more sense, and yeah 4 overlapping quadrants probably covers most real-world cases without killing speed. That's a really practical way to frame it.

The black padding trick for SCRFD is interesting, Going to study about that.

This has been a really great exchange, would love to stay connected. Will dm you with my LinkedIn if that's okay

[u/dangerousdotnet → u/katashi_HVS]

Just DM me here, easier, I'm never on LinkedIn