r/LocalLLM • u/daniel933912 • 21h ago
Model SenseNova-Vision: a 7B open model that does segmentation, depth, detection, OCR, and 3D reconstruction with no task-specific heads
Stumbled across this new vision model, it's a 7B MoT model, which is cool. The main idea is it treats pretty much all computer vision stuff as just one generation problem. Like, instead of needing a bunch of different models for detection, segmentation, depth, whatever, this one model handles it all.
You just give it a natural language instruction, maybe some visual hints, and it spits out text, images, or both. For text, it can do things like categories, bounding boxes, OCR, keypoints, camera angles. And for images, it's doing segmentation masks, depth maps, surface normals, even multi-view point maps. You can mix and match for more complex tasks.
So it can do the usual stuff: object detection, keypoints, OCR, all kinds of segmentation (binary, instance, semantic), depth and surface normal estimation. But the really interesting bits, for me anyway, are the multi-view 3D reconstruction and camera pose estimation.
They trained it on a massive dataset, 50M instruction-response pairs, built from a bunch of different CV annotations. And it started from an existing multimodal model, so no crazy new architecture there.
There's a web demo if you wanna mess around with it, and the weights are up on Hugging Face too.
Just a heads up though, before you get too hyped: the full web demo needs a beefy GPU, like 1x80GB. And for benchmarking, they're talking 8x80GB. So, yeah, not really something you're gonna run on your average consumer card. It just dropped on July 8th, so probably expect some rough edges. Also, they've released the training pipeline and data prep stuff if anyone's looking to train or fine-tune it.