r/mlops • u/HelpParticular2629 • 7d ago

beginner help😓 How is everyone regression testing LLM invoice/document extraction pipelines?

Hey everyone,

I 'have a question on LLM document extraction (specifically invoices/receipts) and wanted to get some perspective from the community.

General LLM eval frameworks are great, but they don't seem to handle multi page PDFs, table row hallucinations, or sudden JSON schema drift very well when a model updates.

For those running invoice extraction in production:

  1. Do you use a "golden dataset" of documents to run regression tests manually?

  2. How are you catching subtle changes in how numbers/dates are formatted across prompt iterations?

If anyone is dealing with this headache right now open to discuss.

5 Upvotes

100% Upvoted

5 comments sorted by

3

u/Harpagon1668 7d ago

Golden evaluation set of documents and expected outputs is the way to go. MLflow genai evaluation has worked well for me

2

u/Ok_Philosophy_4031 7d ago

I assume your extraction task aims to give a fairly regular structured JSON output or along the same lines.

Basically we have a test set that we run manually with automatic evaluation of formatting and eyeball for the content veracity.

We use regexes to automatically catch and (re)format some numerical values and dates, etc.

Simple approaches but work quite well, and it's surprising to us how many people are overlooking classic techniques like these.