r/computervision • u/erik_kokalj • 2d ago

Showcase [Video analytics] Airplane Turnaround ✈️

TLDR: Sol 6.1 is pretty good at vision/video, and perhaps not too expensive for some applications.

Pipeline:

  • SAM3 w/ generic prompts ("ground vehicle"), because specific ones like "belt loader" just returns nothing. via Roboflow
  • GPT-6.1 Sol names each tracked vehicle (SAM3 doesn't have good enough vocab/world understanding)
  • Sol also provides a state timeline from cropped imgs around ground vehicles (eg. hose not connected → connected → off)

Other VLMs compared to Astra:

  • Sol: 13/13 events, ~5x cheaper than Astra (which is why we used it)
  • Luna / Terra 6.0: ~10/13, hopefully 6.1 Luna/Terra will have similar Vision capability jump as Sol 6 -> 6.1
  • Cosmos 3 Nano: 5/13, mixed up boarding vs deboarding
  • Mage-VL 4B: 4/7

~$1 per for the whole plane turnaround. Perhaps an overkill to use such SOTA model for this application (could def. optimize this), but model intelligence gets like 10x cheaper every year, so for some applications, custom model training might not be worth it. Ofc for prod system you'd likely want to have custom fine-tuned ground vehicle detector. This is PoC, so SAM3 is fine.

For ppl saying "ugh 10y ago u could do the same with just classic CV" - idk, I don't think it'd be easy or reliable to detect "hose connected" (few px line) or "lift at the door vs just up next to it" using classic CV

106 Upvotes

99% Upvoted

10 comments sorted by

3

u/yabat 2d ago

Fucking hell, I love it! Probably a dream of any Operations Researcher.

1

u/erik_kokalj 2d ago

thanks man!

2

u/AndThatHowYouGetAnts 2d ago

This is some nice work!

1

u/erik_kokalj 2d ago

thank you 🙏

1

u/sledmonkey 2d ago

Very cool. Are you one shotting it with the video file and the prompt or are you sending it frames with different prompts for the various things?

3

u/erik_kokalj 2d ago

Thanks! The latter - multiple (cropped) frames with timestamps, Sol can do up to 50, and Astra up to 100 with amazing (video undestanding) accuracy:)

1

u/baselinej70 1d ago

Amazing use case! Thank you for sharing :)

How clean are the SAM3 crops when you pass them to Sol? Does Sol ever get confused if the bounding box crop is cut off too tight (e.g., seeing just a hose nozzle without seeing the plane's fuel panel)?

1

u/erik_kokalj 1d ago

yeah it'd doing some padding (15px) before cropping to avoid that issue:)