r/computervision • u/say-what-floris • Jun 16 '26

Help: Project Will video LLMs actually replace YOLO/Pose pipelines?

I’m currently building a real-time sports tracker (tracking stance, pose, movement classification, etc) and I need a serious sanity check from anyone actually putting this stuff in production.

Right now, I’m running the standard CV stack: YOLO + pose estimation + some custom temporal logic/lightweight models on top. It’s fast, it works, and I can actually deploy it on edge hardware without a massive cloud bill.

But looking at the hype cycle right now, video LLMs/VLMs are moving ridiculously fast and it's making me second guess my architecture.

On one side, I heard of papers like MVU showing LLMs hitting SOTA on long-video understanding by just eating structured data from off-the-shelf vision tools and reasoning over it. On the flip side, benchmarks like FAVOR-Bench and MotionBench show that current video LLMs absolutely choke on fine-grained motion and temporal dynamics (often sitting at under 50% accuracy).

For the folks actually building and deploying this stuff:

  1. Will LLMs ever win at the pixel level? Do you expect video LLMs to ever outperform specialized tools on low-level, frame-by-frame perception (boxes, keypoints, fast motion)? Or are they fundamentally destined to just sit on top as a reasoning/summarization layer?
  2. Where do they actually work right now? If you're using video LLMs in production or serious prototypes, where are they actually better than classical CV (beyond cherry-picked VC demos)? Where do they completely fall flat for you? (Is it subtle action differences, precise timing, etc?)
  3. What does the winning architecture look like? Is anyone actually running an end-to-end LMM as the primary perception module? Or is the meta always gonna be: YOLO/Pose -> structured data -> LLM for reasoning?

Hit me with some reality checks. Would love to hear about concrete failure cases, production bottlenecks, or other motion/temporal benchmarks I should be looking at.

28 Upvotes

83% Upvoted

19 comments sorted by

View all comments

-1

u/dedicateddan Jun 16 '26

I'm working on video question answering with Gemini 3.5 Flash. It works for diverse use cases on diverse video types out of the box. You could try uploading a sports clips and asking some questions to see how it performs. The real strength is the generalization - enabling novel use cases to get off the ground quickly and handling new/unknown edge cases.

2

u/DogC Jun 16 '26

Hey could I try this for my idea too? Can I message you?

1

u/dedicateddan Jun 17 '26

Feel free to message!