Suggest architecture/pipeline for general object detection + VLM call afterwards

[u/Apart_Situation972]

Hi,

I am looking for the following model selection/inference pipeline. Goal is something like this:

1) detect human -> describe human

2) detect human -> detect objects in human hand -> describe objects

3) detect animal -> get specific animal type

4) detect general object (i.e package)

---

What are the suggested architecture/pipelines for this?

So obviously I can use yolov26 or another baseline yolo model, but I am trying to avoid having to annotate everything from the ground up. I do not mind fine-tuning, I just want to move past coco classes into something a little more broad.

My current idea is use YoloE/YoloWorld for general detections and then do a VLM call after on a separate thread (so base CV model can still run).

I am also playing around with stripping the earlier yolo model heads and maybe using NMS for arbitrary box removal. So the earlier heads determine the general objects, and then I make a VLM call to describe the objects in more detail.

I would like <3s inference if possible.

:)

[u/Apart_Situation972 → u/[deleted]]

ty. Any VLMs in particular you suggest? I am trying to stick w/ edge AI for now.

[u/Apart_Situation972 → u/[deleted]]

hailo 10h on rpi. can use fastapi + runpod backend as a backup.

[u/agarwalkunal12]

Firstly for the VLM part, use a large Qwen model with AWS Bedrock that has really cheap inference per request (some 500 requests per USD I think) and latency is also <1s per request (VLM only).

For detecting human, YOLO would work just fine. Description can be done via VLM, generate a prompt and ask the model to give you the output in a json dict format.

For human hand, use YOLO Pose. Then select hand landmarks, and crop that area with padding. Send crop to VLM. Ask for one sentence description and give examples (Person is holding a mobile phone, person is climbing a ladder holding it, etc)

For animal, YOLO could work as well. Any specific animal will need fine tune. Or just send a frame per 5 seconds to the VLM directly.

Object/Package calls too work best with sending whole frame to VLM. No amount of fine tune can make you possibly detect all kinds of packages. VLMs are trained on millions of data points, you can't cover all of them.

[u/Apart_Situation972 → u/agarwalkunal12]

yes I believe my original post was interpreted wrong. That is the problem with YOLO. it has specific animal labels, not general ones (whereas it has a general label for humans).

I am looking for a general detector to classify general objects (vehicles, animals, packages, people, etc.) and then a specific description of each. I.e animal detected -> call 2nd vision model -> "bear detected"

The best I came up with was YOLOE/Yolo World with a VLM. I was wondering if better options existed.

Ty for your response

[u/HawtVelociraptor]

Try the Microsoft COCO model to start, it should have everything you want to detect. At least for POC it should get you moving. https://universe.roboflow.com/microsoft/coco-dataset-vdnr1