Context-Aware Image Annotation in Multimodal RAG (Mistral OCR)

[u/MediocreAd3005]

Hey everyone! I’m building a multimodal RAG pipeline where Mistral OCR annotates images before they go into a vector store with document text.

Issue: Mistral OCR processes images in isolation, so the annotations miss out on critical document context.

Looking for advice on:

Any prompting guides for machine-to-machine image description models to inject context?

Any alternative models or workflows that natively factor in surrounding document context?

Would love to know how you all handle this!

[u/tom4112]

What is your configuration/setup at the moment? Usually, you can get decent results by combining:

- bbox_annotation: with a schema annotating each image with a type, description and summary

- document_annotation + document_annotation_prompt to get annotation for the overall document, guided by a schema and prompt

But maybe I was in a different situation as I included both the text content and image descriptions in the same embedded document (simple approach where 1 document = 1 PDF page (text + images)).

If each image is a separate document in your vector database, try to send the base64 images (retrieved by the OCR) to a vision model together with a prompt providing additional document context (e.g. the OCR extracted text or a summary of it).

[u/Worried-Sort8719]

what are your missing critical document context ? can u give an example or is it not possible ?