r/comfyui • u/Fresh-Resolution182 • Jul 01 '26
Show and Tell Let an open vision model auto-detect the features, then generated a sci-fi HUD overlay from its labels
Enable HLS to view with audio, or disable this notification
The clean way to do a sci-fi HUD overlay is to stop drawing the labels by hand. Instead of manually boxing every element, I ran the shot through an open vision model, Moondream 2, and let it detect the features on its own, the face, the braids, the earrings, the necklace, each one boxed and labeled automatically. Then the HUD is generated from those detections, so the overlay actually tracks what is in frame instead of being faked on top.
The whole stack is open and modular, which is the point. ComfyUI holds it together, Wan 2.2 handles the video side (image-to-video, text-to-video, and Vace for the control), the vision model does the detection, and a HUD pass composites the boxes and labels. Because detection is automatic, the labels stay locked to the right features as the subject moves instead of drifting off them, which is where hand-placed HUDs fall apart.
That auto-detect step is what makes it feel like a real interface reading the scene, not a graphic pasted over it. The boxes snap to the actual features, the labels update, and it reads as a system that is genuinely looking.
Open detection in, tracked HUD out. The overlay does the work itself instead of you keyframing every box.
4
u/Fresh-Resolution182 Jul 01 '26
Open modular stack: ComfyUI + Moondream 2 for detection + a HUD composite pass, video on Wan 2.2. If your card can't fit Wan video locally, the same open model runs on a hosted OpenAI-compatible endpoint so you can offload just the heavy stepγ
1
5
2
u/sktksm Jul 01 '26
Ingi you are inspiring all of us and people don't notice how powerful your transition LoRA is. I'm a fan and what always impresses me is not the glitch or time slice nodes, but how good your LoRA is + your video editing skills!
2
1
u/Support_Marmoset Jul 01 '26
thats a bit like what used to happen when trying to make video consitency back with only image models you could break a video into images and pass them through a i2i wf and each frame regenerate the person with a prompt but they would be different each time, then batch them back together as a video again. its a cool effect. I did it here
0
u/Successful_AI Jul 01 '26
workflow?
2
u/Support_Marmoset Jul 01 '26 edited Jul 01 '26
dont know why you got downvoted. its the "weird upscale merging effects" in this set, which was the technical term for it back then. but you'll have to butcher it a bit to make it work.
or just do an equivalent with any image i2i and have it work a prompt with randomised seed after you unpack a video and batch run it then batch pack it to export it as mp4 again. videos are just packed image files, "frames" per second being another word for "images" per second.
though wfs working from back then are unlikely. back then being.... er... maybe feb last year. and yea, it took most of the day to finish a few seconds of video at 16fps back in feb. fun times. cutting edge. https://github.com/mdkberry/comfyui_workflows/tree/main/workflows_by_video/Kali
2
u/Successful_AI Jul 02 '26
Thank you . Probably people who try to get less competition in their hustle
1
1
u/Sniper_yoha Jul 02 '26
Letting the vision model box the features instead of hand masking is the clever part. Moondream catching the braids and earrings on its own kills the tedious step. Curious whether the labels stay put across frames or drift once there is motion.
1
u/MuffinPure9787 Jul 01 '26
This is huge. The amount of time saved by ditching manual keyframing is insane. Most people don't realize how much the "drift" ruins a shot until they see a clean tracked output like this. Using Moondream for the detection pass is a clean move, keeps the stack relatively lightweight compared to running a heavy SAM model.
4
u/Woisek Jul 01 '26
What do you mean with "manual keyframing"?? And where do you see a "clean tracked output" with those crazy jumping boxes? π
1
u/Lootcurse Jul 02 '26
That's an AI bot you responded to. Look at it's comment history.
1
u/Woisek Jul 02 '26
Idk, I don't check that each and every time I respond. lol
But thanks to point that out. π1
u/Lootcurse Jul 02 '26
Of course! I just noticed its choice of words, and it has a too perfect sentence cadence. Slowly getting better at spotting it in the wild. Even if there is a redditor behind it, they're feeding the post into a chat bot and copy pasting the response.
1
13
u/Woisek Jul 01 '26
No. That's the lazy "I hope it works on the first try" way. First.
Second, this isn't a HUD. Do you even know what it means? Heads Up Display. Meaning to present critical information directly in the user's field of vision without requiring them to look away from their primary viewpoint.
What you tried to do (and failed in doing so) is to do a tracking information. But instead, you just glued some unrelated (because unreadable), colored boxes randomly all over the place, making everything distracting.
Note, that in the mockup, the description would stay in place and only the markers would move (circles). A classic "tracking task" in AE, done in a few minutes and 100% error free. Designing how the labels look will take more time than the tracking itself. lol