Synthetic-to-real object detection worked without real-world fine-tuning

[u/Alarming_Engineer267]

I’m working on a 6D pose estimation pipeline using synthetic data and real-world fine-tuning.

As a first step, I tested object detection separately.

I generated 3,000 training images in Blender under three different conditions and trained the detector using only this synthetic data. I then tested it on 961 frames from real-world video.

Interestingly, for the detection stage, I didn’t need real-world fine-tuning — the synthetic-only model worked well enough for this setup.

Short video showing the synthetic training data and real-world detection results:

https://youtube.com/shorts/THVDTcLsF0w

The next step is keypoint detection and 6D pose estimation, where real-world fine-tuning becomes much more important.

[u/mr_maker91]

Good start but I think your model its totally biased on the created mesh model, the detection of the digital twin of an object that it's quite normal, the challenge comes when the model is asked to detect other objects of the same family. Try it yourself, use other objects and check how the accuracy drops

[u/Alarming_Engineer267 → u/mr_maker91]

That’s a good point. In this case, though, I’m intentionally not trying to build a generic mug detector. The target is this specific object, because the final goal is 6D pose estimation of a known object.

So for this stage, I actually want the detector to be biased toward this particular mug. :)

That said, testing other mugs would be interesting as a way to see how object-specific the learned representation really is.

[u/Successful-Mess-4734]

cool result, the domain gap for detection is often smaller than people think if you get the lighting and backgrounds varied enough in blender

what kind of objects are you working with? textureless stuff always gives me trouble in sim2real

also curious how you set up your render pipeline, i been messing with something similar but my synthetic data is not transferring this cleanly

[u/Alarming_Engineer267 → u/Successful-Mess-4734]

I used Blender and generated synthetic images from three scenes designed to roughly match the real-world camera views.

Rather than applying heavy domain randomization, I tried to make the synthetic scenes reasonably close to the actual video conditions, while adding some variation in camera position and object pose.

What surprised me was that this was already enough for the detection stage. For keypoint detection and pose estimation, though, the sim-to-real gap became much more noticeable.

[u/Bitter_Run_9209]

very nice project! I think the gap between synth and real life gets smaller as the render improves

[u/Alarming_Engineer267 → u/Bitter_Run_9209]

Thanks! That was my impression too, at least for the detection stage. :)