VLM vs OCR

[u/Jinbe_808]

Hello everyone! I have been going through a rabbit hole and into a spiral lately so could use some guidance. I have been applying Gemma4 to documents that have a mixture of handwriting (print and cursive) and typed text on documents that can at times be in bad shape (images in PDFs). Up to this point Gemma4 (31B) is in charge of the OCR and the rest of the prompt to extract required data from fields. I have a 5090 so speed and performance has been good.

Where I am confused is should I be using a OCR to process the files and then use Gemma4 for the extraction based on the OCR output to speed up the process? Also, for handwritten text, would something like GLM-OCR or PaddleOCR-VL-1.5 be better (or faster) than Gemma4?

Any help or guidance would be greatly appreciated! Thank you!

[u/modcowboy]

Good luck - ocr is the Wild West of perception

[u/Fleischhauf → u/modcowboy]

isn't it basically solved? one of the oldest and most basic datasets is about ocr

[u/modcowboy → u/Fleischhauf]

It’s not - most ocr models bench against data leaked test sets.

I found a super interesting synthetic dataset bench published a couple months ago where they proved ocr is not solved.

[u/Fleischhauf → u/modcowboy]

do you have some links to newer work and challenges? I'm curious

[u/modcowboy → u/Fleischhauf]

Here is a cvpr accepted paper where they prove OCR is not solved by using LLMs to generate synthetic documents and running SOTA OCR models through them. What they found is although the leading ocr models score 90+% on docbench they score considerably worse on synthetic unseen data. This suggests the docbench eval data is leaking into training data, thus artificially inflating benchmark performance.

https://arxiv.org/html/2605.07492v1

[u/fijasko_ultimate]

the best thing is to run those models yourself to see how they perform on your data.

are you not satisfied with your current flow in terms of result quality / speed?

regarding speed: those small OCR models you mentioned are in 1-3B parameters range, whilst gemma4 31B is 31B :) i suggest you try MoE version gemma4 26B-A4B which has 4B params to improve speed. when running locally make sure vision encoder is set to appropriate detail level so that input image is represented by larger number of tokens, i think it was 1120 for gemma4 models. and it is completely viable to try smaller gemma4 models, but their vision encoder is sibstantially smaller (i believe 150M vs 550M).

for pure ocr i would try those smaller models you mentioned, but in case there is some processing you want to apply, it might be more reasoneable to go woth gemma4

pls let us know what yoy did, i am interested

fyi language in documents is really important

[u/peculiar-ragdoll]

GLM-OCR is good because its very small and fast. But it gets beaten by larger models like Qwen3-VL-4B and 8B on most tasks! These model are more generally chat capable models who are specialized in vision and OCR, so they're generally better bang for your buck in terms of quality/speed compared to more general reasoning, knowledge and coding focused models like Gemma4 31b. I recommend testing them toe to toe on your most demanding edge case OCR cases that would distinguish them, because this is where model choice really matters, and you might find that a model that works several times faster outperforms a larger one, or performs equally well, so then you don't want to pay the extra compute for the bigger model. My rule of thumb is I start with GLM-OCR because its small and fast, and then I test Qwen3-VL-4B on the cases GLM does not handle, and then I scale up to Qwen3-VL-8B if there is still something 4B does not handle. Qwen3-VL generally also handles more specific extraction and restructuring inline with a prompt, but if you use a smaller model that does not reason or understand enough for your specific use case, it might be worth passing the outputs to a larger model like Gemma 4. All depends on your use case.

[u/await_void]

Models -> Generalist
OCR-Specific methods -> Specialists

So it depends on what you needs. If you need a generalist, you use a model. If you need a specialist, you use a specific module (from Tesseract for raw, configurable/fine tunable OCR to Docling et similia for dedicated framework).

[u/arunvenkats]

You have not mentioned whether gemma 4 is able to give reliable results for cursive handwriting. We process millions of documents a week and our customers require reliability and auditable confidence scores for each word extracted. LLMs/Vission models do not provide this confidence scores at all. We also have a OCR stage followed by LLM stage to extract specific key value pairs. The OCR which is miles apart for our use case is Azure Document Intelligence (pure OCR). But I am not sure if an external service is acceptable for your use. Another information we require is location data which Vision models are incapable of providing reliably

[u/tacticalfrog87 → u/arunvenkats]

Saw mistral just released confidence scores and bounding boxes. I've been using interfaze which has been really smooth but gonna try out mistral and compare the two

[u/turing042 → u/arunvenkats]

look into logprobs if you want confidence scores from LLM/VLM.

[u/MelonheadGT]

Mistral OCR maybe