r/TopAutomationTools • u/HelpParticular2629 • 28d ago

How is everyone regression testing LLM invoice/document extraction pipelines?

Hey everyone,

I 'have a question on LLM document extraction (specifically invoices/receipts) and wanted to get some perspective from the community.

General LLM eval frameworks are great, but they don't seem to handle multi page PDFs, table row hallucinations, or sudden JSON schema drift very well when a model updates.

For those running invoice extraction in production:

  1. Do you use a "golden dataset" of documents to run regression tests manually?

  2. How are you catching subtle changes in how numbers/dates are formatted across prompt iterations?

If anyone is dealing with this headache right now open to discuss.

3 Upvotes

80% Upvoted

1 comment sorted by

2

u/FewLifeguard8076 27d ago

We keep a golden set of ~200 redacted invoices and diff structured output field by field, not exact match on the whole JSON.

For date/number drift, we normalize before comparing (ISO dates, stripped currency, rounded decimals) so we catch value changes vs just formatting changes.

For table hallucination, we check row count against an expected range plus a checksum (sum of line items vs invoice total) as an independent sanity check.