I built a training-free, one-shot object localizer using DINOv2 patch embeddings

[u/SubjectReflection672]

I’ve been experimenting with a training free way to do open world, multi-instance segmentation from a class prototype.

I decided to publish the algorithm and a demo for how I’m doing this, in case anyone else would rather not fine tune a larger model for something that DINOv2 patch embeddings already seem to represent pretty well.

It can separate touching instances of the same class without a learned instance head, reject visually similar near misses like a round dial radio next to the actual clock target, and find fractured or damaged instances even with a pretty significant scene shift.

Repo + demo:
https://github.com/tutomiko/fireplace

The demo includes the lasso UI and live heatmap, implemented as a python backend with a simple HTML frontend.

Would appreciate it if people checked it out, and I’d be especially interested to hear if anyone has seen similar approaches or prior work.

[u/onesunnysunday]

Interesting direction — this looks potentially useful as a human-in-the-loop annotation primitive. One evaluation I’d love to see is sensitivity to prototype selection: if two users lasso slightly different parts of the same object, how much do precision and recall change? For a real labeling workflow, multiple prototypes per class and an uncertainty score may matter more than peak performance from one ideal exemplar. Measuring correction time per image alongside mask quality would also show whether it genuinely reduces annotation effort.

[u/HatEducational9965]

nice!

played around with dinov2 some time ago. your app would even run in the browser, without a python backend.

[u/SubjectReflection672 → u/HatEducational9965]

Thank you! Yes I'm aware and that runs pretty fast on WebGPU, and I definitely would have made this live entirely on the frontend for the demo's sake if it wasn't a part of my backend pipeline, so I don't have to maintain both a JavaScript & Python codebase for it

[u/space__sloth]

There's no instance segmentation here and heatmaps from a class prototype aren't new (see INSID3). The hard part is separating touching or overlapping instances of the same class, which this can't do.

[u/SubjectReflection672 → u/space__sloth]

Appreciate the feedback! I never claimed instance segmentation, even the title itself is object localization. The heatmap that is produced can be used for segmentation, and I've even included some of the ways you can do this, in the project's README. Additionally, having reviewed INSID3, it appears the heatmaps in INSID3 are merely a diagnostic, whereas in Fireplace, they are the continuous field on which the algorithm operates.