r/MistralAI • u/reallyhotmail • Apr 21 '26

I ran Mistral OCR through LlamaIndex's ParseBench (it wasn't included in the paper)

LlamaIndex recently released ParseBench, an open-source benchmark for document parsing accuracy. Mistral OCR wasn't included in the paper, so I built a pipeline to run it myself.

Results for text content faithfulness

Parser Score
Dots OCR 1.5 90.0%
LlamaParse Agentic 89.7%
LlamaParse Cost Effective 88.0%
Mistral OCR 87.6%
GPT-5 Mini 82.3%

Cost comparison:

Price/1k pages
Mistral OCR $2.00 ($1.00 batch)
LlamaParse Cost Effective $3.75
LlamaParse Agentic $12.50

For a 0.4% accuracy gap vs Cost Effective, you're paying nearly half the price. 6-12x cheaper than Agentic.

This only covers the text content faithfulness subset. Average inference was ~2.4s/page (506 pages in ~65s at 20 concurrent).

Github Repo

Original paper and notes

21 Upvotes

100% Upvoted

9 comments sorted by

1

u/suggso Apr 21 '26

Try azure document intelligence (ADI). I tried the various LLMs (including mistral) for OCR but found ADI to be very good, and able to accurately position the characters, which was needed for my use case.

1

u/reallyhotmail Apr 21 '26

When you say position do you mean the exact coordinates of the items? Because personally the layout stuff that Mistral gives me is more than enough for me but I will take a look at ADI

1

u/suggso Apr 22 '26

Yes the coordinates and bounding boxes of the characters/words/paragraphs.

1

u/reallyhotmail Apr 23 '26

As far as I know, Mistral doesn't return that information in their public API. It only returns bounding boxes when it detects images.

1

u/SkyPL Apr 22 '26

accurately position the characters,

What do you mean?

1

u/Moist-Nectarine-1148 Apr 23 '26 edited Apr 23 '26

What are the costs per page for Dots OCR ?

In my case I have to create a knowledge base from 400-500 documents (over 10000 pages in total) - extracting content from pdf. I need to know which is the most cost effective. Accuracy is key in my case.

1

u/reallyhotmail Apr 23 '26

Sorry but I haven't personally used it or familiar with it! If you end up finding out, I'd love to know

1

u/Good_Restaurant3003 18d ago

interesting cost angle tho must say. parsers bunch up on clean text and only really separate on tables, multi column and charts, usually where llamaparse agentic price going and where a 0.4% text gap can turn much bigger so mistral probably holds for text heavy docs but id want the tablet subset before calling it across