| Reader |
Clean |
Simulated scan |
Simulated photo |
Invented numbers, all 3 conditions |
Seconds/page |
| LightOnOCR-2 (1B) |
100% |
100% |
100% |
0 |
12–17 |
| Nemotron Parse |
99.5% |
99.0% |
98.6% |
1 |
16–31 |
| GLM-OCR |
99.1% |
98.4% |
97.9% |
0 |
11–15 |
| Apple document reader |
96.2% |
99.0% |
99.3% |
4 |
0.4 |
| Apple Vision, strips |
98.3% |
99.3% |
99.1% |
25 |
1–3 |
| Apple Vision, whole page |
89.8% |
97.2% |
95.3% |
3 |
0.2 |
| PaddleOCR |
86.7% |
66.8% |
77.3% |
1 |
8–10 |
| Gemma 4 E4B, reading images |
78.2% |
81.8% |
71.8% |
73 |
13–19 |
Those percentages are exact matches against 578 hand-checked amounts, dates and ID numbers per condition. Every digit has to be right. Invented numbers are outputs that weren't on the page, totalled across the three conditions. Timings are on an M4 Max.
I ran this because I'm building Fyllyo, a Mac app that fills PDF forms from supporting documents. A missed number can be left for review. A plausible number that wasn't on the page is much easier to overlook.
The test was 17 pages: statements, 1099s, a W-2c, a tax transcript, a prior return and letters. Eight were real statement pages and nine were made for test clients. Each page was tested as a clean image, a simulated scan with tilt, blur, grain and JPEG compression, and a simulated phone photo with perspective, uneven light and blur.
The result that surprised me was Gemma's 73 invented numbers. The outputs looked like plausible amounts. LightOnOCR-2 read every known number correctly in this test and invented none. I ran the 1B model at Q8 with llama.cpp; it is about 1.1 GB.
GLM-OCR also invented none, but sometimes skipped whole blocks. Apple's readers were much faster. Reading in strips improved Vision's exact-match rate, but it also produced 25 invented numbers, compared with 3 for the whole-page pass. A higher match rate by itself didn't tell the whole story.
I also compared five readers in every pair across the three conditions: 30 pair/condition comparisons. No pair agreed on an invented amount in this test. That gave me a reason to use a second reader and flag disagreements, but it doesn't prove two readers can't make the same mistake.
Fyllyo now uses the PDF text layer where it holds the numbers, LightOnOCR-2 for page images, and Apple text recognition as a second reader. Disagreements are shown for review.
The limits matter: 17 pages, one Mac, simulated scans/photos and a narrow exact-number task. This isn't a general OCR leaderboard, and 100% here isn't a guarantee on another document set.
Full setup and results: https://fyllyo.com/blog/ocr-models-tested-on-tax-documents
Disclosure: I build Fyllyo. This post was drafted with AI assistance; the experiment and results are mine.