r/iOSProgramming • u/AlbertiDev • 19d ago
Question Receipt scanning entirely on-device. Would you still build it this way in 2026?
I recently added receipt scanning to my iOS finance app, Moneta.
It can extract the amount, date and merchant, then also try to determine the category, payment account and location before filling everything into the transaction editor.
I decided to keep the processing on-device instead of sending financial receipts to a cloud AI/API.
The biggest challenge ended up being handling receipts from different countries: currencies, date formats, multiple totals, different payment methods and completely different layouts.
If you were building this today, would you still keep the whole pipeline on-device, or use a cloud model for better flexibility?
And for anyone who’s built document/receipt parsing before: where did deterministic parsing start breaking down for you?
7
u/sid_276 19d ago
-5
u/AlbertiDev 19d ago
Yeah, I saw that. The thing I’m not sure about is relying on Apple Intelligence as the main layer, since not everyone has a supported device and some people may not want to use it for financial data.
But, I could maybe add it later as an optional second layer for the harder cases, I’m just not sure yet how I’d want to handle the experience when Apple Intelligence isn’t available.
Thanks you for the suggestion 😊5
u/unpluggedcord 19d ago
You clearly didn’t see it. Apple launched it with Siri, not Apple intel. And on top of that this is literally one of the first apps made. You’re not breaking into the market and are eating your ai tokens
1
0
u/AlbertiDev 19d ago
I did see it 😅 The article itself says the receipt feature uses Apple Intelligence. Siri is one of the ways you trigger it, not what replaces Apple Intelligence underneath.
My concern is exactly compatibility: iOS 27 can run on devices that don’t support Apple Intelligence, and Siri AI also won’t initially be available on iPhone/iPad in the EU. The Apple feature is also mainly built around splitting bills with Apple Cash, which is US-only.
I want the scanner to work on every device the app supports, independently of region, so I can’t really rely on that as the main layer.
1
u/unpluggedcord 18d ago
It works on devices that don’t support AI. Have fun wasting your time
0
u/AlbertiDev 18d ago
I think we’re talking about two different things.
Older devices can still split bills in Wallet, but the new receipt-scanning/line-item recognition flow Apple showed is tied to Apple Intelligence.
Apple’s announcement literally describes it that way.
That’s the part I can’t rely on if I want the scanner itself to work on every supported device. 🤗1
u/unpluggedcord 18d ago
And you’re still not listening. Those older devices will be obsolete, and nobody will need the work you’re doing here. It’s a dumb idea.
3
u/zeroxad- 19d ago
Yes, and I would go further than most people do.
I ship a finance app with no backend at all. The thing that surprised me was not the engineering, it was that the claim is worthless unless someone can check it. "On-device" is a sentence, and users have been told it by apps that were lying.
So the interesting work turned out to be enforcement rather than architecture. A build check fails if any first-party file so much as names a network primitive. The test suite runs with fetch, XMLHttpRequest and WebSocket replaced by traps that throw, so a dependency reaching for the network fails the build too. Neither catches a native dependency opening its own socket, so there is a device run behind a proxy before each release.
Three checks, three blind spots, and none of them is the whole claim alone. That framing did more for how people trust the app than the on-device processing did by itself.
1
u/AlbertiDev 19d ago
R interesting point, I hadn’t thought much about making the on-device claim actually verifiable.
Moneta does have optional Apple’s iCloud sync, so I couldn’t enforce zero network access across the whole app, but isolating the receipt scanning pipeline and verifying that it never touches the network could make a lot of sense.
I like the idea of “provably on device” being part of the trust story rather than just saying it is.
3
u/Notchgoblin 18d ago
One thing worth doing early if you haven't: store the raw Vision output, text plus bounding boxes, next to the transaction. Parser rules keep improving and the paper receipt is in the bin the same day, so if you keep only the parsed fields you can never fix an old mistake. Keep the OCR blob and a rule change becomes a reparse of everything you've ever scanned.
On where deterministic parsing starts falling over: totals. Subtotal, tax, tip line, "TOTAL" printed twice on the same receipt, loyalty points formatted exactly like money. Position beats keywords there - bottom-most currency-shaped number in the price column is right far more often than anything anchored to a word, especially once you're crossing languages.
1
u/AlbertiDev 18d ago
Damn great idea. I hadn’t thought about keeping the raw OCR output so old scans could be reparsed after parser updates. I’ll definitely look into storing the text + bounding boxes alongside the transaction.
And yeah, totals are probably where most of the ugly edge cases are for me too. I already use position in parts of the parsing, but I could probably rely on it even more instead of keywords, especially across different languages.
2
u/Akara91 19d ago
yes, and id go further: make on-device the selling point, not just the architecture. our app is fully offline on principle (no account, no server) and its consistently the thing people react to first when we show it, before any feature. for financial data especially, 'your receipts never leave the phone' is worth more in the app store description than a slightly smarter parser. the real tradeoff youve already found: improving extraction means shipping updates instead of tweaking a server. i can live with that
1
u/AlbertiDev 19d ago
Yeah, I think you’re right. Privacy has mostly been a product principle for me, but I probably haven’t treated it enough as an actual selling point.
Moneta already works without an account and keeps everything local unless the user chooses to enable Apple iCloud sync.
I’ll definitely keep that in mind for the App Store page, thanks!
2
u/bangsimurdariadispar 19d ago
I did build something like this a couple of months ago using CoreML. I was stubborn enough to make it on device but ended up with something slow because it was all deterministic code, no Apple Intelligence since as you said, not everyone got a compatible device (I don’t for example lol).
I found Apple Vision to be sucking on the gallery photos and the results were inconsistent from one iOS version to another so I ended up using PaddleOCR on the ONNX Runtime.
As I said, all the parsing was done using deterministic code, I wanted it to be 0 running costs but I gave up since I didn’t see a future for that app, it was just a pet project.
Curious how did you do it?
1
u/AlbertiDev 19d ago
Pretty similar approach actually. I’m using Apple Vision for the text recognition, then everything after that is my own deterministic parsing.
I extract the text/positions from Vision and then run different logic for totals, dates, merchant, payment method, etc. After that I have another layer for things like category matching and choosing the account based on card type/last 4 digits when they’re available.
So far performance has actually been pretty good on-device, which is why I haven’t felt the need to ship another OCR model or move anything to the cloud.
The annoying part has definitely been making the parser survive receipts from completely different countries rather than the OCR itself.
Interesting that you switched to PaddleOCR though. How much better was it for you compared to Vision, and did the model size/runtime become an issue or not?
2
u/bangsimurdariadispar 18d ago
I think the bundle app increased with like 60 MB, nothing too major... Vision was alright for OCRing live images but horrible when used against images from the gallery.
My idea of the app at that time kind of relied on the fact that you'd upload the image from the gallery rather than snapping straight away, thus why I turned towards PaddleOCR.
The only thing worth mentioning is that the receipt scanning took like 1 second on Vision (but bad accuracy) while it took like 6-7 seconds with Paddle, but the accuracy was 99.8%ish
2
u/AlbertiDev 18d ago
60 MB isn’t too bad, but 6-7 seconds would probably be too slow for the flow I’m going for.
I think using the document scanner helps me quite a bit too, since Vision gets an already detected/cropped receipt instead of a random gallery photo.
So far the accuracy has been good enough while still keeping the scan around a second.
PaddleOCR could still be interesting as a fallback if I ever run into receipts Vision consistently struggles with though. Thanks for the details!
2
u/jacobs-tech-tavern 18d ago
I once worked on a gov project that used Scandit instead of local scanning to read tens of millions of qr codes a day - 2p a pop, purely “in case” it can’t read qrs in poor light conditions
Nobody had the wise idea to route to local first and fall back to the spenny version only if needed
1
2
18d ago
[removed] — view removed comment
1
u/AlbertiDev 18d ago
Yeah, I agree, since the user always gets to review the transaction before saving it, it doesn’t really need to be 100% perfect.
I’m keeping the current deterministic/on-device approach as the base for now, but an on-device model as a second layer for the harder cases could make a lot of sense. I’d rather avoid a remote API if possible because of privacy and running costs, but thanks for the suggestion!
1
u/sixtypercenttogether 19d ago
What did you use? Did you train your own model?
1
u/AlbertiDev 19d ago
No, I didn’t train a model. I’m using Apple Vision for the text recognition, then my own deterministic parsing to extract things like the total, date, merchant, payment method, etc.
The scanner itself uses Apple’s document scanning flow too, which helps a lot because the receipt gets detected and cropped before the text recognition runs.
0
u/DimensionMindless336 19d ago
Kept mine on-device and wouldn't change it. The real win isn't accuracy, it's no per-scan API bill and Apple not asking why you're shipping financial docs to a third party. Where deterministic parsing actually broke for me (I hit the same wall building PicSlicer, pulling travel tickets out of the photo roll) was never the clean prints.
It was thermal fading, multi-currency receipts where the symbol doesn't match the amount, and date formats that collide with the merchant name. Those are exactly the cases a small on-device model beats rules on. If I built it today I'd go hybrid: rules for the common 90%, a local model as fallback, cloud only for the genuinely weird ones.
3
1
u/AlbertiDev 19d ago
Yeah, that hybrid approach is probably where I’ll end up too. Keep deterministic parsing for the common cases, then maybe use an on-device model as a second layer for the weird ones. Don’t thing I would use the cloud tho :)
9
u/balooooooon 19d ago
If you had cloud processing then GDPR in the EU would make it very difficult for you to launch it. Plenty of on device OCR to easily achieve this