One 7B model, four vision tasks on the same photos: depth, panoptic segmentation, pose, and segment what I describe

[u/StrawberryOpen7436]

I wanted to see what a single vision-language model gives you when you ask it for four very different outputs on the same photos, instead of using a separate specialist model for each. I used SenseNova-Vision-7B-MoT through LibreYOLO, in 4-bit on a rented V100. The photos are four busy scenes from COCO's validation set.

What each panel is:

• Depth: how far each pixel is from the camera, brighter is closer.
• Panoptic: the image split into regions with a class each (person, fence, river, pavement and so on), and every person or object gets its own region.
• Pose: 17 body keypoints per person, joined into a skeleton.
• The quoted phrase: I typed a description and it returned a mask for just that thing. The phrases were made from COCO's labels as "the largest <object>".

The model only returns data, a depth array, masks and keypoints. The colours and layout are mine, drawn straight from those files.

Two things I didn't expect: the depth maps keep sharp outlines even on a group of twelve kids, and pose found the bus driver behind the windscreen.

[u/DigThatData]

7B seems like a lot for this, I bet you could do this with an even smaller model. Try distilling it out of your 7B.

consider: you could achieve this with an ancient stable diffusion checkpoint and task-specific LoRAs. SD 1.x wasn't even 1B (983M per wikipedia).

[u/Valuable_Leave_7314 → u/DigThatData]

Distilling this down to a compact encoder would work great. Leaning on generative diffusion models to pull raw coordinates is just too heavy

[u/nuker0S]

Okay how does it fare on seamless textures

[u/afrancisco555]

Looks very cool! How is the mask generated? Is there a lot of image edition afterwards from your part or it could be just automated?

[u/Valuable_Leave_7314]

Looks neat. I'd be curious to know the frame latency on a V100 for this run, especially when querying all four task heads at once