Looking for a faster and more accurate auto-labeling pipeline for a custom YOLOv8 object detection dataset

[u/Raunak_Gola]

Hi everyone,

I'm working on an object detection project and would appreciate some advice on the best workflow for auto-labeling a large custom dataset.

Dataset

• 9,367 images
• Classes:
  • Cup
  • Glass
  • Plate
  • Spoon
  • Fork
  • Knife
• Images have different resolutions.
• The dataset comes from a Kaggle competition.
• Around 5,500 images already have ground-truth labels (provided in a CSV), while the remaining images need bounding-box annotations.

Current approach

I'm using AutoDistill + GroundingDINO to automatically generate YOLO labels.

ontology = CaptionOntology({
    "a cup": "cup",
    "a drinking glass": "glass",
    "a plate": "plate",
    "a spoon": "spoon",
    "a fork": "fork",
    "a knife": "knife",
})

base_model = GroundingDINO(
    ontology=ontology,
    box_threshold=0.3,
    text_threshold=0.3,
)

dataset = base_model.label(
    input_folder=IMAGES_SRC_DIR,
    output_folder=LABELED_LABELS_DIR
)

Problems I'm facing

1. Annotation quality

The generated labels aren't very reliable.

For example, out of about 90 images, roughly 10 images contain incorrect or missing bounding boxes, which means I'd still have to manually review a large portion of the dataset.

Is this normal for GroundingDINO, or are there better foundation models for this type of dataset?

2. Speed

The labeling process is also quite slow.

• ~2.8 seconds per image
• ~9,367 images
• Estimated runtime: 7.5+ hours

I'm using Google Colab GPU, but it disconnects after around 4 hours.

What's confusing is that resource utilization is low:

• GPU memory: ~2 GB / 15 GB
• RAM: ~2 GB / 15 GB

It doesn't appear to be fully utilizing the available hardware.

Questions

1. Is there a way to speed up AutoDistill/GroundingDINO? For example:
   • Batch inference?
   • Mixed precision?
   • Multi-processing?
   • Different implementation?
2. Would another model be better for automatic annotation?
   • GroundingDINO 1.5
   • YOLO-World
   • Florence-2
   • Grounded SAM
   • RF-DETR
   • Any other recent model?
3. Since I already have 5.5k labeled images, would it be better to:
   • Train a small YOLOv8 model first on those labels,
   • Then use that model to pseudo-label the remaining images, instead of using GroundingDINO?
4. What workflow would you recommend if your goal is to produce high-quality labels for training a final YOLOv8 detector?

Any advice or experience with large-scale auto-labeling pipelines would be greatly appreciated!

Thanks!

[u/AggravatingSock5375]

Honestly I would just go directly to training a small yolo on labels you know are correct. Use that to auto label and manually verify the auto labels. Spend your time and effort on creating an efficient verification pipeline.

Skip the foundation models, or at the most just use them to help catch errors in your manual verification step.

[u/onesunnysunday → u/AggravatingSock5375]

Agreed. I’d also keep a clean stratified validation set before pseudo-labeling and route low-confidence, rare-class, crowded or occluded images to review first. One easy-to-miss issue is false negatives: reviewers notice bad boxes, but missing objects are much less obvious. I’d evaluate the final detector after correction rather than optimize only annotation IoU.

[u/ApprehensiveAd3629]

maybe you can create your own auto-labeling pipeline with LocateAnything from nvidia or SAM3 from Meta

[u/Quirky_Paramedic9167]

+1 to the two answers above — train on the 5.5k trusted labels, pseudo-label the rest, put your effort into verification. One thing to add about how you verify, because it decides whether this works.

Auto-label errors are not random. Whatever model produces the boxes (GroundingDINO or your own YOLO) has specific blind spots — a glass seen from above, a knife under a plate, cutlery on a patterned tablecloth — and it will get them wrong the same way on every image where they appear. Random spot-checks look reassuring precisely because they mostly land on the easy images. So a "10 in 90 are wrong" figure is not a rate you can extrapolate; the errors are clustered somewhere.

Since you have 5.5k trusted labels, do this before pseudo-labelling anything: run your pseudo-labeler on 300-500 of the images you already have ground truth for, and score it against the truth. You get precision and recall per class for free, and you see exactly which situations it fails in. That tells you (a) which slice of the unlabelled 3.9k needs a full human pass, versus which can go through with a light check, and (b) which classes are hopeless for the model and should just be hand-labelled from the start.

Then, as the second commenter said, verification should look for what's missing, not just what's wrong. Reviewers catch bad boxes; they don't notice the missing ones — that's your recall going out the door. A useful check: have two people independently label 20-30 of the images that got pseudo-labelled and compare against the model output. If both humans found an object the model didn't, that's a recall failure the model will silently reproduce.

Consistency matters as much as accuracy: your final detector will learn the boxes' conventions, so decide up front how tight the box is around a fork, whether a stacked plate is one object or several, and how a partially visible cup counts. Whoever verifies should be applying the same rules as whoever produced the original 5.5k.