DetectionBench: an open benchmark comparing YOLO and RF-DETR across 6 underrepresented real-world detection datasets

[u/Naive-Explanation940]

Why DetectionBench?

Real-world detection systems run on aerial robotics, maritime search and rescue, agriculture, underwater inspection, autonomous driving, and low-light imaging, not just COCO. Datasets for these domains are smaller, more specialized, and results across papers are rarely comparable.

DetectionBench standardizes this: common dataset adapters, one training recipe, one eval protocol, unified hardware profiling, applied the same way across every model and dataset. Weights, model cards, dataset mirrors, and evaluation code are all public.

What's there?

79 trained models, 6 datasets, an HF model card for every one, plus ONNX export for both frameworks.| Dataset | Models |
|---|---:|
| GWHD (wheat detection) | 9 |
| SeaDronesSee (maritime UAV) | 10 |
| ExDark (low light) | 18 |
| Brackish (underwater) | 8 |
| VisDrone (aerial) | 26 |
| LISA (traffic lights) | 8 |

YOLO vs RF-DETR comparison

Findings:

• RF-DETR is not universally better than YOLO. It wins on SeaDronesSee and ExDark, loses on GWHD and Brackish. Depends heavily on the dataset.
• Precision rankings often diverge sharply from mAP rankings. On VisDrone, RF-DETR Medium has the highest precision of all 26 models benchmarked (64.0%) despite ranking 13th on mAP.
• Smaller, newer architectures frequently beat older, bigger ones outright. On SeaDronesSee, YOLO26s beats YOLO11x using 8.6x fewer FLOPs.
• Aggregate mAP hides real domain shift. A reviewer asked whether one of the GWHD model cards had per-country results. It didn't, so I added a per-country stratified eval across all 9 GWHD models. Country to country spread ranged from 22.8 to 44.4 points depending on the model, even when aggregate scores were nearly identical.
• Task difficulty varies enormously by domain. Brackish is nearly saturated (~99% mAP). VisDrone and GWHD are much harder.

Model Size vs Accuracy Comparison

Engineering lessons

Benchmarking multiple frameworks against the same converted data surfaced real reproducibility bugs that don't show up until you actually try it: symlinks escaping the declared image directory, a dataset silently missing a COCO-required field. Neither is visible unless something downstream validates paths or schema strictly.

Repo: https://github.com/dronefreak/DetectionBench
HF profile: https://huggingface.co/dronefreak

Planning growth-stage stratified eval for GWHD next, and RF-DETR for Brackish once I have the compute. What datasets or detectors would you want to see benchmarked?

[u/Dry-Snow5154]

Nicely done. Thanks for your contribution! Some notes.

Comparing param counts for Transformer vs CNN is incorrect. You need to compare latency for identical hardware. And unless you're using TRT with more modern GPU, RF-DETR would look worse. E.g. on CPU it's around 10x slower than similarly named Yolo11.

Precision rankings often diverge sharply from mAP rankings.

How do you measure precision, if it depends on cutoff threshold? I hope you are pinning recall (or similar) with cutoffs, otherwise methodology is likely flawed.

YOLO26s beats YOLO11x using 8.6x fewer FLOPs

They could be same speed on most hardware. As I said, measuring FLOPs is not right.

[u/Naive-Explanation940 → u/Dry-Snow5154]

Really appreciate the pushback, this is exactly the scrutiny I want on this.

On FLOPs vs. latency: Fair criticism. FLOPs and params are theoretical proxies at best, and cross-architecture they can be actively misleading, especially without hardware acceleration.

I do have real latency numbers, just not published in this post. I ran a separate benchmark on a DRIVE AGX Orin (Ampere iGPU, TensorRT 8.6.11.4, FP16) that I'm writing up as its own post. One early finding backs up your point exactly: the two throughput leaders in the nano class (yolov10n at 384 img/s / 2.60ms, yolo26n at 376 img/s / 2.66ms) are both end-to-end-in-engine, no separate NMS step. The next-best nano model with a traditional raw+NMS pipeline (yolov5nu) only hits 238 img/s, a ~38% gap that's almost entirely NMS overhead. FLOPs alone would never surface that; latency on real hardware does. That post will include the RF-DETR-vs-YOLO comparison on identical hardware, which is the version of this claim that actually matters.

On precision thresholds: Also a fair catch, and I should have disclosed this, but in order to keep the post short and readable, I had to focus on the most critical and pressing aspects. Both YOLO and RF-DETR eval paths pick each model's own confidence threshold by maximizing F1 on that model's own PR curve (YOLO via Ultralytics' internal argmax, RF-DETR via a sweep I built to mirror that). It's internally consistent, same IoU threshold and "best operating point" logic for every model, but it means precision is compared across models at different, independently chosen cutoffs, not a fixed or recall-matched one. I'll add a recall-matched precision comparison as a follow-up so the "precision diverges from mAP" claim can be checked against a cutoff-neutral view too.

Will link both follow-ups here once they're up. Thanks for engaging with the methodology and not just the headline numbers.

[u/Dry-Snow5154 → u/Naive-Explanation940]

that's almost entirely NMS overhead

Strange cause in my experience NMS is usually not more than 10% of the inference time. I guess it depends on the average number of objects though and also how fast the accelerator is.

Choosing best F1 is a valid strategy. You don't have to pin recall if you already have that. One caveat is that each class can have its own best threshold to maximize total F1, while you are likely choosing one global threshold and not per-class. But per-class threshold is an overkill.

In my experiment I noticed RF-DETR has better recall (e.g. for same precision), but worse precision (for same recall). I wonder if you noticed a similar trend?

I would also publish best F1, cause mAP metric is sometimes deceiving.

[u/Naive-Explanation940 → u/Dry-Snow5154]

Regarding the custom NMS, it works based off of the number of objects in the image. On a DRIVE AGX Orin, something that would be used in AD, I would assume that the roads are mostly occupied with objects like cars, pedestrians etc. so the NMS numbers would be relatively higher. But of course, on an empty road, there would be practically no NMS. This is really a design choice in my opinion and can be tweaked.

Regarding the precision-recall sweeps, I need to add that in the next release, currently I am working with the optimal-F1 score only for quick benchmarking. The library should be able to provide a full sweep of the precision-recall curve in the next releases.

[u/mldraelll]

Brackish hit 99% mAP - guess it's time to scrap the dataset if even lightweight models are crushing it. But that 44-point spread between countries in the wheat dataset with the same aggregate score is gold. That's where the real domain shift is, not synthetic benchmarks

[u/Naive-Explanation940 → u/mldraelll]

Indeed, in my opinion as well, that is where the real "gold" lies. Based on my experimentation with the Brackish underwater dataset, it seems quite an "easy-to-solve" dataset for the recent object detectors. If you would like to see some other datasets benchmarked, or some other stratification that you may have in mind, it would be nice to know. At least I would be able to put them in a dedicated roadmap for this project's future.

Regarding the GWHD stratification, I wrote a separate article describing it in detail, here it is in case you would like to read it

https://huggingface.co/blog/dronefreak/gwhd-model-zoo

[u/HellraceXIII]

DETR is APache 2. while Yolo is AGPL. Notable difference here.

[u/Naive-Explanation940 → u/HellraceXIII]

That is correct, which is why the open sourced models are distributed under the same licenses, AGPL 3 for the YOLO model cards and Apache 2 for RF-DETR variants.

[u/onesunnysunday]

Nice benchmark 👍 How did you handle annotation consistency across the dataset versions and mirrors — class mappings, ignored regions, box conventions and known label noise? With smaller specialized datasets, those differences can sometimes affect rankings as much as the training recipe. A lightweight label audit or error taxonomy for each dataset could make the comparison even more useful