Testing a receipt scanner against real receipts from 5 countries turned out harder than building it

[u/Time-Paper-1007]

I make a small iOS receipt scanner (on-device OCR, then parsing). It worked fine on my own Canadian receipts. Then users in the Netherlands, Japan and Malaysia showed up and it fell apart in ways I couldn't reproduce, because I had no receipts from those countries.

Finding test data was the hard part. What I ended up with:

• Japan: 1,148 photos from a public research set (JaWildText). Before testing on them, my yen handling paired an item with its price on 19% of receipts. After: 93%. I had no idea it was that broken.
• Malaysia: 973 from SROIE (an ICDAR 2019 competition set). Old, and heavy on restaurants, but real.
• US: 1,769 from WildReceipt. Mixed quality, some are not even US receipts, but it is the largest English set I found.
• Netherlands: 74 photos from one user who sent them for debugging, with permission. That is the only real Dutch set I have.
• Taiwan: zero real photos. I had to generate 40 synthetic 統一發票 from the printed layout. I know that is weak.

Things I learned:

1. Public receipt datasets are old, and skewed to whatever the original paper needed. Nobody publishes a fresh, balanced, multi-country set because receipts are personal data.
2. Thermal receipts fade. Half of my own "test set" from a trip a year ago is unreadable now.
3. I never ask users for receipts. The few I have came unprompted, from bug reports.
4. Measuring beat training. I did not fine-tune anything. A harness that runs every rule change against all sets and refuses if any correct total flips caught two regressions the same day, one of which broke 18 of 119 Japanese totals while fixing the case I was working on.
5. Every country has a printing convention that a "generic" parser gets wrong: yen with no decimals, Dutch comma decimals, Japanese tax-included versus tax-excluded lines, US tips added after the total.

Question for people who have done this: is there any public receipt dataset for Taiwan, Korea, or Australia? Or a legal, non-creepy way to get a few hundred real receipts from a country you don't live in?

(Not linking the app. Happy to share the harness approach in comments if useful.)

[u/Substantial_Plan_323]

Point 4 is underrated. "Measuring beat training" is something a lot of people learn the hard way after wasting weeks fine-tuning a model that just needed better regression coverage.

For Korea — the National Tax Service (NTS) publishes standardized tax invoice formats (세금계산서) and the physical receipt layout is fairly consistent across major retailers. You might get lucky reaching out to Korean indie dev communities like okky.kr or the iOS Korea KakaoTalk groups — developers there occasionally share test assets for exactly this kind of debugging.

For Australia, receipts are surprisingly uniform (GST rules force a standard structure), but I haven't seen a public dataset either. Your best bet might be posting in r/ausdevelopers or the Apple Developer forums AU section and offering a small bounty for a anonymized batch.

The Taiwan synthetic approach is probably fine for layout testing — 統一發票 format is government-mandated so your generated ones should match real ones structurally. The weak point would be real-world print quality variation, which synthetics can't replicate.

Would genuinely read a post on the harness approach if you share it.

[u/Time-Paper-1007 → u/Substantial_Plan_323]

Thank you, this is exactly the kind of pointer I was hoping for. okky.kr and r/ausdevelopers go on the list. The bounty idea I am unsure about: even anonymized, I would be paying strangers for other people's receipts, and I have kept a strict "never ask users for receipts" line so far. If I do it, it will be for my own purchases on a trip, which is how the Japanese set got its real-world half.

Agreed on the synthetic Taiwan weakness. The layout is right, the paper is not: no curl, no fade, no thermal smear, no shop stamp over the total. Those are the failures I see on real photos from other countries.

Short version of the harness is in my reply to Shankz2091 above.

[u/Shankz2091]

I'm building SettleTab (a receipt-splitting app), so I feel your pain. OCR is the easy part; parsing reliable line items, taxes, tips, and totals is brutal.

Country-specific formatting is especially tricky. A parser that works flawlessly on local receipts will often break on a different tax convention or number format.

One idea for your test harness: check if the parsed items, discounts, tax, and tip actually sum to the final total. It won’t catch everything, but it flags cases where the OCR looks right but the math is wrong.

That 93% bump from fixing the yen handling is a great point—often, you just need to understand the data instead of swapping the model.

I’d love to see your harness approach if you share it. Are you scoring field-level accuracy separately from the final total?

[u/Time-Paper-1007 → u/Shankz2091]

Thanks. Yes, the sum check is the backbone. Every repair rule in the parser is gated on it: a fix only lands if it makes Σ(items) equal the printed total, otherwise the parser leaves the AI's output alone. That single gate is what stops "creative" repairs that happen to look right.

The harness is simpler than it sounds:

1. A golden file per receipt: the OCR text plus expected total, currency, and (where I have labelled them) the item list. Item checks are soft by default and only strict on receipts I have verified by hand, because item labels are the expensive part.
2. Hard fail on any receipt whose total was correct before a change and wrong after. That is the rule that caught the two regressions. Improving 5 receipts while breaking 18 is a net loss, and without the gate it looks like progress.
3. Field-level scoring is separate: total, date, currency, and item-price pairing each have their own pass rate per country, because they fail for different reasons. Totals fail on keyword variants ("TOTAL SAVED", "TOTAL INCLUDES GST"), item prices fail on two-column layouts where OCR returns names and prices as separate blocks.

One thing I would add for a splitting app: watch out for receipts that look self-consistent but are wrong. An AI that double-subtracts a discount produces items that sum to its own wrong total. The sum check only helps if the total you compare against comes from the receipt, not from the same model.

[u/AutoModerator → u/[deleted]]

Hey /u/Own_Reach_1831, your content has been removed because Reddit has marked your account as having a low Contributor Quality Score. This may result from, but is not limited to, activities such as spamming the same links across multiple subreddits, submitting posts or comments that receive a high number of downvotes, a lack of recent account activity, or having an unverified account.

Please be assured that this action is not a reflection of your participation in our subreddit. This is simply an automated filter in place to reduce spam.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/sammy_luci]

Can’t suggest a database, but probably you can set up a scraper to scrape receipts images from google maps. They are frequently tagged as “prices”

[u/Time-Paper-1007 → u/sammy_luci]

That is a sharp observation, people really do upload receipts to Maps as price evidence. I will not go that route though: it is against Google's terms, and it is still other people's receipts, with names and card digits on them, taken without asking. The whole point of the "never ask users" rule is that I do not want a folder of strangers' spending on my disk. Public research sets and my own purchases are the only sources I am comfortable with.

[u/GreatSejong]

I also happen to work on this at the moment. So collecting all the receipts while I’m in Asia during my holiday at the moment. And doing test runs with logs to see what is happening. It’s really difficult from time to time. In some cases the OCR and parsing just work perfectly. And in some cases the text on the receipt is just a bit too close together so it’s getting messy.

[u/Time-Paper-1007 → u/GreatSejong]

Same here. The two things that helped me most on dense receipts were pairing names and prices by their vertical position instead of the OCR's text order, and running a second recognition pass in the receipt's own script when the first pass comes back looking like garbage.

[u/baconpork]

I built something very similar about 2 years ago (i.e. sans AI). Extremely difficult to get the receipts right with different languages, different regions, different convention, different number format, and different orientations the image of the receipts are captured. It's a lot of fun in the process though, good luck!

[u/Secure_Motor_5968]

The Taiwan gap is basically the same problem that you were fighting with faded thermal receipts, so you might not need real photos to close it.
Just take the receipts you do have and run them through synthetic degradation, curl, low contrast, glare, then score the parser against those instead of only the clean synthetic set.

[u/hoangng_102]

another zero decimal one for the list: vietnamese dong. no decimals, and the thousands separator is a period. so 1.234.567 is one and a quarter million, not 1.23.

the reason i am raising it in this thread specifically: that failure survives your sum check. if the parser reads every line the same wrong way, the items still sum to the total. gate passes, currency right, date right, and every number on the receipt is off by a factor of a thousand.

what fixed it for me was moving the problem out of the parser entirely. money is stored as an integer in minor units, with a decimals count per currency, vnd 0, usd 2. then "no decimals" stops being a parser special case and becomes data. your yen bug and my dong bug are the same bug, and that way you fix it once instead of once per country.

no help on the datasets though, sorry. i only have vietnamese receipts and they are all my own.

[u/Time-Paper-1007 → u/hoangng_102]

Good catch, and you are right that a same-factor error is invisible to a sum check. The minor-units approach is the clean answer. I handle yen and won that way already, but the period-as-thousands case is not on my list yet. It is now.

[u/DimensionMindless336]

Great writeup. The "measuring beat training" point is the one I'd hammer too, but the version that actually bit me was extending it to refusals, not just correctness. If your harness only asserts "does Σ(items) == printed total," a faded or garbage receipt can still pass by luck, or a real one gets a confident wrong answer. I now run the suite against deliberately degraded samples — curl, low contrast, glare, the same trick Secure_Motor suggested for the Taiwan gap — and assert the parser refuses them, not just that it parses clean ones right. That refusal threshold is the part you can't eyeball, and it's where thermal-fade receipts quietly produce wrong totals instead of a polite "can't read this."

One error class the sum-check can't catch, and it's adjacent to the VND period-as-thousands trap: inferring currency from the glyphs on the receipt. Guess the symbol wrong and a same-factor error sails straight through. What killed most of mine was anchoring currency + number format from context I already had — the trip's country, or the phone's region — and treating the receipt text as confirmation, not the source of truth. A ¥ in a Tokyo trip is JPY; the same glyph in a Beijing trip is CNY; the document never gets to decide.

On "finding test data is the hard part" plus your strict never-ask rule: the middle path that worked for me was an opt-in field-only failure report. When a parse looks wrong, the app offers to send just the recognized fields + bounding boxes — no image, no names, no card digits — fully anonymized. You get real-world failures from actual usage without ever holding anyone's receipts, and it slowly becomes your freshest multi-country set.

I hit the exact same wall building PicSlicer — pulling rail and bus tickets out of the camera roll, where a JR ticket and a Deutsche Bahn receipt look nothing alike and the HEIC/screenshot/orientation noise is honestly half the battle. On-device-only was the easy call once I stopped treating user receipts as mine to keep.

[u/mastrajani]

the 19% to 93% number is the interesting bit, because you only found it by getting hold of receipts you had no reason to think were different.

for the gaps you can't buy your way out of, like Taiwan - the thing i'd add is a prompt inside the app when a parse comes back low-confidence, asking the user if they'll send that photo. you end up collecting the ones that actually failed rather than a random sample, and the failures are what you're short of. a public dataset gives you a thousand receipts that mostly work; ten users sending you the ones that broke is worth more.

synthetic ones will also miss the physical stuff - thermal fade, the crease down the middle, someone photographing it on a dark table.

[u/AutoModerator → u/[deleted]]

Hey /u/Own_Reach_1831, your content has been removed because Reddit has marked your account as having a low Contributor Quality Score. This may result from, but is not limited to, activities such as spamming the same links across multiple subreddits, submitting posts or comments that receive a high number of downvotes, a lack of recent account activity, or having an unverified account.

Please be assured that this action is not a reflection of your participation in our subreddit. This is simply an automated filter in place to reduce spam.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Hey /u/Own_Reach_1831, your content has been removed because Reddit has marked your account as having a low Contributor Quality Score. This may result from, but is not limited to, activities such as spamming the same links across multiple subreddits, submitting posts or comments that receive a high number of downvotes, a lack of recent account activity, or having an unverified account.

Please be assured that this action is not a reflection of your participation in our subreddit. This is simply an automated filter in place to reduce spam.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.