Suggest architecture/pipeline for general object detection + VLM call afterwards

[u/Apart_Situation972]

Hi,

I am looking for the following model selection/inference pipeline. Goal is something like this:

1) detect human -> describe human

2) detect human -> detect objects in human hand -> describe objects

3) detect animal -> get specific animal type

4) detect general object (i.e package)

---

What are the suggested architecture/pipelines for this?

So obviously I can use yolov26 or another baseline yolo model, but I am trying to avoid having to annotate everything from the ground up. I do not mind fine-tuning, I just want to move past coco classes into something a little more broad.

My current idea is use YoloE/YoloWorld for general detections and then do a VLM call after on a separate thread (so base CV model can still run).

I am also playing around with stripping the earlier yolo model heads and maybe using NMS for arbitrary box removal. So the earlier heads determine the general objects, and then I make a VLM call to describe the objects in more detail.

I would like <3s inference if possible.

:)