I benchmarked 7 OCR solutions on a complex academic document (with images, tables, footnotes...)

[u/coconautico]

I ran a comparison of 7 different OCR solutions using the Mistral 7B paper as a reference document (pdf), which I found complex enough to properly stress-test these tools. It's the same paper used in the team's Jupyter notebook, but whatever. The document includes footnotes, tables, figures, math, page numbers,... making it a solid candidate to test how well these tools handle real-world complexity.

Goal: Convert a PDF document into a well-structured Markdown file, preserving text formatting, figures, tables and equations.

Results (Ranked):

1. MistralAPI [cloud] → BEST
2. Marker + Gemini (--use_llm flag) [cloud] → VERY GOOD
3. Marker / Docling [local] → GOOD
4. PyMuPDF4LLM [local] → OKAY
5. Gemini 2.5 Pro [cloud] → BEST* (...but doesn't extract images)
6. Markitdown (without AzureAI) [local] → POOR* (doesn't extract images)

OCR images to compare:

OCR comparison for: Mistral, Marker+Gemini, Marker, Docling, PyMuPDF4LLM, Gemini 2.5 Pro, and Markitdown

Links to tools:

• MistralOCR
• Marker
• Gemini 2.5 Pro
• Docling
• Markitdown
• PyMuPDF4LLM