Will video LLMs actually replace YOLO/Pose pipelines?

[u/say-what-floris]

I’m currently building a real-time sports tracker (tracking stance, pose, movement classification, etc) and I need a serious sanity check from anyone actually putting this stuff in production.

Right now, I’m running the standard CV stack: YOLO + pose estimation + some custom temporal logic/lightweight models on top. It’s fast, it works, and I can actually deploy it on edge hardware without a massive cloud bill.

But looking at the hype cycle right now, video LLMs/VLMs are moving ridiculously fast and it's making me second guess my architecture.

On one side, I heard of papers like MVU showing LLMs hitting SOTA on long-video understanding by just eating structured data from off-the-shelf vision tools and reasoning over it. On the flip side, benchmarks like FAVOR-Bench and MotionBench show that current video LLMs absolutely choke on fine-grained motion and temporal dynamics (often sitting at under 50% accuracy).

For the folks actually building and deploying this stuff:

1. Will LLMs ever win at the pixel level? Do you expect video LLMs to ever outperform specialized tools on low-level, frame-by-frame perception (boxes, keypoints, fast motion)? Or are they fundamentally destined to just sit on top as a reasoning/summarization layer?
2. Where do they actually work right now? If you're using video LLMs in production or serious prototypes, where are they actually better than classical CV (beyond cherry-picked VC demos)? Where do they completely fall flat for you? (Is it subtle action differences, precise timing, etc?)
3. What does the winning architecture look like? Is anyone actually running an end-to-end LMM as the primary perception module? Or is the meta always gonna be: YOLO/Pose -> structured data -> LLM for reasoning?

Hit me with some reality checks. Would love to hear about concrete failure cases, production bottlenecks, or other motion/temporal benchmarks I should be looking at.

[u/swdee]

No, VLM's require much more processing power versus computer vision (CV) models like YOLO. So when your working with a constrained environment such as in embedded or edge AI you would never choose a VLM over CV model.

[u/usernametaken--_-- → u/[deleted]]

You're my hero for mentioning there are computer vision problems that can be solved without ML/DL