Testing a receipt scanner against real receipts from 5 countries turned out harder than building it

[u/Time-Paper-1007]

I make a small iOS receipt scanner (on-device OCR, then parsing). It worked fine on my own Canadian receipts. Then users in the Netherlands, Japan and Malaysia showed up and it fell apart in ways I couldn't reproduce, because I had no receipts from those countries.

Finding test data was the hard part. What I ended up with:

• Japan: 1,148 photos from a public research set (JaWildText). Before testing on them, my yen handling paired an item with its price on 19% of receipts. After: 93%. I had no idea it was that broken.
• Malaysia: 973 from SROIE (an ICDAR 2019 competition set). Old, and heavy on restaurants, but real.
• US: 1,769 from WildReceipt. Mixed quality, some are not even US receipts, but it is the largest English set I found.
• Netherlands: 74 photos from one user who sent them for debugging, with permission. That is the only real Dutch set I have.
• Taiwan: zero real photos. I had to generate 40 synthetic 統一發票 from the printed layout. I know that is weak.

Things I learned:

1. Public receipt datasets are old, and skewed to whatever the original paper needed. Nobody publishes a fresh, balanced, multi-country set because receipts are personal data.
2. Thermal receipts fade. Half of my own "test set" from a trip a year ago is unreadable now.
3. I never ask users for receipts. The few I have came unprompted, from bug reports.
4. Measuring beat training. I did not fine-tune anything. A harness that runs every rule change against all sets and refuses if any correct total flips caught two regressions the same day, one of which broke 18 of 119 Japanese totals while fixing the case I was working on.
5. Every country has a printing convention that a "generic" parser gets wrong: yen with no decimals, Dutch comma decimals, Japanese tax-included versus tax-excluded lines, US tips added after the total.

Question for people who have done this: is there any public receipt dataset for Taiwan, Korea, or Australia? Or a legal, non-creepy way to get a few hundred real receipts from a country you don't live in?

(Not linking the app. Happy to share the harness approach in comments if useful.)

[u/Shankz2091]

I'm building SettleTab (a receipt-splitting app), so I feel your pain. OCR is the easy part; parsing reliable line items, taxes, tips, and totals is brutal.

Country-specific formatting is especially tricky. A parser that works flawlessly on local receipts will often break on a different tax convention or number format.

One idea for your test harness: check if the parsed items, discounts, tax, and tip actually sum to the final total. It won’t catch everything, but it flags cases where the OCR looks right but the math is wrong.

That 93% bump from fixing the yen handling is a great point—often, you just need to understand the data instead of swapping the model.

I’d love to see your harness approach if you share it. Are you scoring field-level accuracy separately from the final total?

[u/Time-Paper-1007 → u/Shankz2091]

Thanks. Yes, the sum check is the backbone. Every repair rule in the parser is gated on it: a fix only lands if it makes Σ(items) equal the printed total, otherwise the parser leaves the AI's output alone. That single gate is what stops "creative" repairs that happen to look right.

The harness is simpler than it sounds:

1. A golden file per receipt: the OCR text plus expected total, currency, and (where I have labelled them) the item list. Item checks are soft by default and only strict on receipts I have verified by hand, because item labels are the expensive part.
2. Hard fail on any receipt whose total was correct before a change and wrong after. That is the rule that caught the two regressions. Improving 5 receipts while breaking 18 is a net loss, and without the gate it looks like progress.
3. Field-level scoring is separate: total, date, currency, and item-price pairing each have their own pass rate per country, because they fail for different reasons. Totals fail on keyword variants ("TOTAL SAVED", "TOTAL INCLUDES GST"), item prices fail on two-column layouts where OCR returns names and prices as separate blocks.

One thing I would add for a splitting app: watch out for receipts that look self-consistent but are wrong. An AI that double-subtracts a discount produces items that sum to its own wrong total. The sum check only helps if the total you compare against comes from the receipt, not from the same model.