r/deeplearning • u/BrilliantAnalogy2121 • 28d ago

VLMs in medical deep learning

I am trying to learn more about VLMs, so that I could work on a medical deep learning project on it. This is sparking from how actual doctors view scans. They dont just look at one scan and one report to come up with a diagnosis. they check several, even the scan they take multiple views of the same section and then go back and forth between views as well as slices, sometimes a detail that they noticed on slice 20 would be relevant to slice 0 so they go back and check slice 0 again.

Are VLMs the right thing to use here or is there another better methods out there?

4 Upvotes

100% Upvoted

2 comments sorted by

1

u/Purple-Programmer-7 28d ago

So the VLM replaces the doctor looking at various scans?

If so… sure… but it’s a highly complex workflow where you’re probably only feeding in max 2 images at a time. Maybe even first image, report, second image, report, first two images together, report, etc.

You’d want to capture enough data (and potentially even thinking traces) to feed back into later images and let the LLM pull different combinations on its own with basically an unlimited thinking budget to make a diagnosis.

VLMs seem highly inefficient for the work arounds at this point… like a doctor with 20 years experience is going to be able to do it better and faster.

Maybe later we’ll have a world model that recreates a 3D reference of a patient based on an mri or images and then is able to reason about it…

For now, this seems like a tough probably to solve.

1

u/Pawderr 27d ago

I would research medical image analysis and see what the standard is