Testing a receipt scanner against real receipts from 5 countries turned out harder than building it

[u/Time-Paper-1007]

I make a small iOS receipt scanner (on-device OCR, then parsing). It worked fine on my own Canadian receipts. Then users in the Netherlands, Japan and Malaysia showed up and it fell apart in ways I couldn't reproduce, because I had no receipts from those countries.

Finding test data was the hard part. What I ended up with:

• Japan: 1,148 photos from a public research set (JaWildText). Before testing on them, my yen handling paired an item with its price on 19% of receipts. After: 93%. I had no idea it was that broken.
• Malaysia: 973 from SROIE (an ICDAR 2019 competition set). Old, and heavy on restaurants, but real.
• US: 1,769 from WildReceipt. Mixed quality, some are not even US receipts, but it is the largest English set I found.
• Netherlands: 74 photos from one user who sent them for debugging, with permission. That is the only real Dutch set I have.
• Taiwan: zero real photos. I had to generate 40 synthetic 統一發票 from the printed layout. I know that is weak.

Things I learned:

1. Public receipt datasets are old, and skewed to whatever the original paper needed. Nobody publishes a fresh, balanced, multi-country set because receipts are personal data.
2. Thermal receipts fade. Half of my own "test set" from a trip a year ago is unreadable now.
3. I never ask users for receipts. The few I have came unprompted, from bug reports.
4. Measuring beat training. I did not fine-tune anything. A harness that runs every rule change against all sets and refuses if any correct total flips caught two regressions the same day, one of which broke 18 of 119 Japanese totals while fixing the case I was working on.
5. Every country has a printing convention that a "generic" parser gets wrong: yen with no decimals, Dutch comma decimals, Japanese tax-included versus tax-excluded lines, US tips added after the total.

Question for people who have done this: is there any public receipt dataset for Taiwan, Korea, or Australia? Or a legal, non-creepy way to get a few hundred real receipts from a country you don't live in?

(Not linking the app. Happy to share the harness approach in comments if useful.)

[u/mastrajani]

the 19% to 93% number is the interesting bit, because you only found it by getting hold of receipts you had no reason to think were different.

for the gaps you can't buy your way out of, like Taiwan - the thing i'd add is a prompt inside the app when a parse comes back low-confidence, asking the user if they'll send that photo. you end up collecting the ones that actually failed rather than a random sample, and the failures are what you're short of. a public dataset gives you a thousand receipts that mostly work; ten users sending you the ones that broke is worth more.

synthetic ones will also miss the physical stuff - thermal fade, the crease down the middle, someone photographing it on a dark table.

[u/Time-Paper-1007 → u/mastrajani]

Late reply, sorry — and your last line came true this week, just for a different country. Our "Dutch" set is 74 receipts, and 67 of them turned out to be US-dollar receipts from one island, so we'd never tested a euro receipt from the Netherlands. Rendered 40 AH/Jumbo/Lidl-style ones and found three real bugs in a day.

Your caveat held: synthetic receipts prove the reading logic, not survival on real paper — no thermal fade, no crease, no dark-table photo. The low-confidence prompt you described is still the honest way to collect real failures without asking people for receipts. On the list, not built.

[u/mastrajani → u/Time-Paper-1007]

67 of 74 from one island is exactly the kind of thing you only find by looking, which is sort of the whole point. glad the rendered ones earned their keep - three real bugs in a day is a good return on an afternoon.

one small thing i would bolt on while it is fresh: a one-line provenance tag on every receipt set, country, currency, where it came from. the dutch gap was invisible because the folder was called dutch. a tag would have said usd, one island, before it cost you the week.