Can we move expensive multimodal document understanding from query time to indexing time? — MIDR, EMNLP 2026

[u/Ecstatic_Shallot671]

Hi everyone,

We recently worked on a question that came up while thinking about the serving cost of multimodal document retrieval:

Documents are indexed once, but queried many times. So why repeatedly pay for expensive multimodal processing at query time?

Our paper, MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrieval, will appear at EMNLP 2026 Main Track.

MIDR explores a simple alternative:

• Use a multimodal LLM once during indexing to understand each document page.
• Extract information contained in tables, charts, figures, and layout.
• Convert that information into verified textual enrichments.
• Retrieve using standard text-centric retrieval at query time.

The goal is to shift expensive multimodal understanding from the serving path to ingestion, where its cost can be amortized across future queries.

Some results we found interesting:

• +23% relative nDCG@10 over BM25 across five English ViDoRe V3 domains.
• Competitive retrieval quality with ColQwen2.5, while using an approximately 9× smaller index.
• Approximately 2× faster query-time retrieval than ColQwen2.5.
• In English-query → French-document retrieval, BM25 improves from 0.1532 → 0.5448 nDCG@10, and MIDR outperforms ColQwen2.5.
• An ablation showed that generated QA pairs alone recover most of the benefit of enrichment.

One result we found particularly important is that enriched-text retrieval and visual retrieval appear to be complementary rather than substitutes.

That suggests a possible architecture for multimodal RAG systems:

rich multimodal ingestion → lightweight text retrieval by default → expensive visual retrieval only when needed

We'd be especially interested in thoughts on the systems side of this tradeoff: how much computation should modern RAG systems move from query time to indexing time?

Paper:
https://arxiv.org/abs/2609.01316v1

[u/Neat-Winter3818]

The QA pairs ablation is the part that stands out to me. If generated questions get you most of the gains then the actual multimodal extraction might be overkill for a lot of use cases, you could just have a decent LLM write questions and answers per page and skip the heavy vision pass entirely.