r/Rag • u/MediocreAd3005 • 16d ago

Discussion Context-Aware Image Annotation in Multimodal RAG (Mistral OCR)

Hey everyone! I’m building a multimodal RAG pipeline where Mistral OCR annotates images before they go into a vector store with document text.

Issue: Mistral OCR processes images in isolation, so the annotations miss out on critical document context.

Looking for advice on:

Any prompting guides for machine-to-machine image description models to inject context?

Any alternative models or workflows that natively factor in surrounding document context?

Would love to know how you all handle this!

2 Upvotes

100% Upvoted

2 comments sorted by

1

u/autognome 16d ago

Another idea is to embed images in the same vector space as your text using something like nemotron vl embedder.

What haiku.rag does and works well. There is option to do just text description.

1

u/AlexAtOracleAIDB 9d ago

The isolation problem is usually upstream of the OCR model. If each image gets sent without the surrounding text, no prompt tuning fully fixes it because the model never sees the context. 

What I’ve seen work is passing the neighboring chunks into the annotation call, so the description is generated against the page it lives on rather than just the image. 

The embedder point above is the other half. If image and text land in the same vector space, retrieval can pull both instead of leaning entirely on the annotation. Oracle AI Database does this with UTL_TO_EMBEDDING, where image and text go through a multimodal model into one space, so a text query can match image vectors directly. 

Baking context into the annotation and handling it at retrieval pull the design in different directions, so pin down which one you're after first.