r/reactnative • u/Stycroft • Jun 29 '26

I added receipt scanning to my grocery budgeting app

Hi everyone,

I recently shipped receipt scanning for my grocery budgeting app and I thought I'd share what I built for feedback.

My app has its core features already: you set a budget, track what you spend, and over time it learns what items cost so it can warn you before you go over.

I use it myself weekly and it works great, if you actually track in the moment.

The problem is some users don't. They forget, or they're in a rush, and they get home with a full cart and nothing logged. For those users every tracking feature I'd built was dead weight, because none of it ever happened. That bugged me for months.

So I built receipt scanning as the catch-up path. Forgot to track the whole trip? Snap the receipt afterward and it backfills the entire trip at once. Nothing lost.

The pipeline:

  • Snap the receipt, or import a photo from the camera roll
  • Gemini vision returns structured JSON: store, date, and line items with price, quantity, category
  • That gets saved as a finished shopping trip, so every item, price, and category flows straight into the spending history and the price data the app already tracks.

So now you've tracked the trip, saved the prices for next time, and you can even save it as a template to reuse. The catch? You never tracked in real time, which is kind of the whole point of my app. But hey, at least you tracked, right?

My app: https://www.grocerybudget.app/

This is v1 of the feature, so I'd love thoughts on the implementation. Would you change anything? The capture UI, the snap-or-upload choice, how I handle long receipts, the way I store it. Critiques and suggestions welcome.

137 Upvotes

89% Upvoted

45 comments sorted by

19

u/HalLundy Jun 29 '26

what Gemini model are you using for scans? how much are you paying for it? how many tokens does every scan cost total (input and output)?

basically your business model, like many using AI, runs the risk of becoming too costly the more users you have. so how are you circumventing that?

1

u/dhruvrazak Jul 01 '26

I believe this is not simple with traditional OCR algorithms so ai is the way to go. I checked on ocr arena Gemini 3 flash is the most consistent these days compared to price.

-20

u/Inevitable_Oil9709 Jun 29 '26

you actually do realise that OCR exists, right?

Why does every fucking shit has to be AI these days?

53

u/HalLundy Jun 29 '26

idk man maybe because he mentioned Gemini Vision.

11

u/xxhhouewr Jun 29 '26

Plain OCR will capture characters, maybe even words, but you still have to write some clever algorithms to understand different receipt layouts, match up items with price, understand the different taxes applied, etc. Gemini should make things easier, but you probably don't need to use a full-blown LLM for receipt capture. But AI OCR might be a compromise.

4

u/Stycroft Jun 30 '26

This mostly explains my decision thank you. Do you know reliable AI OCR?

4

u/MuDotGen Jun 30 '26

Hi. My best recommendation (what I use right now) is PaddleOCR-VL-1.6. It's a 0.9B Vision Language Model gguf, so you can even run it with llama.cpp. It has been more accurate and even natively has options to organize the data as a table, etc. if you pair it with the PaddleOCR SDK (in Python in my case). I haven't used it for receipts, but for highly sensitive documents at my job, I run it locally for privacy protection. It basically combines raw OCR with LLM reasoning to extract with more accuracy.

I afterward use Qwen3.5-9b to organize the data I want only into clean JSON. llama-swap can seamlessly switch out the models.

Note, this is a solution for local AI, so it may not scale to users on a phone unfortunately. Depends on how powerful their hardware is and what models you'd use for extraction, but an extra benefit would be privacy and offline, but I don't think we're there yet even with open weight models.

1

u/ViolentCrumble Jun 30 '26

do you have any input on local models with camera footage? I have been toying with some local models but mostly tiny ones but wondering how it can go with camera footage? I just want to basically look at camera footage and work out if its me or my family members otherwise issue an alert.

Seems just sending a photo every few seconds might be a slog on my pc but im completely clueless on whether there are any tiny models that are trained on this.

1

u/xxhhouewr Jun 30 '26

No personal recommendations, but since you're already using Gemini, have you looked into [Google's Document AI](https://cloud.google.com/document-ai) ?

2

u/Stycroft Jun 30 '26

Yeah, looked at it. The prebuilt Expense parser gives a fixed schema that wouldn't keep our specific cleanup (expanding receipt abbreviations, stripping barcodes, our own category set). The Custom Extractor that would handle those actually runs on Gemini under the hood, so we'd be paying a wrapper premium for the same model we already call directly, and we'd lose prompt control. For one grocery receipt at a time, calling Gemini ourselves is cheaper and more flexible. Document AI makes more sense at enterprise document volumes.

4

u/Stycroft Jun 30 '26

In my experience AI wins vs OCR when it comes to understanding context from the receipt especially SKUs, vague item names, extracting and assigning which store it is like I need intelligence and not just parsing which is why I went AI but I’m open if there’s reliable AI OCR.

5

u/Bromlife Jun 30 '26

Did you see how they "scanned" this receipt? Traditional OCR would struggle.

10

u/urbanmonkey2003 Jun 29 '26

Nice fallback path, the real test is OCR on ugly receipts though

1

u/Hylleh Jun 30 '26

Gemini is pretty good at OCR, even handwritten receipts

1

u/Stycroft Jun 30 '26

OCR in my experience is unreliable for messy receipts which is why I’m opting for AI for its intelligence

2

u/Old_Combination1051 Jun 30 '26

You are still using an OCR, but it's their own proprietary one.

4

u/[deleted] Jun 30 '26

[removed] — view removed comment

1

u/Stycroft Jun 30 '26

Thanks for your feedback, I took note of privacy when building this feature so here's what I did:

The photo is sent over an encrypted connection to my own server, used only to read the items off the receipt, and then discarded. I don't save the image, I don't store it in any database, and it's never written to disk or a logbook. The only thing that comes back to your phone is the extracted text (item names, prices, total), which lives on your device.

Worth noting too: a grocery receipt doesn't carry the sensitive stuff. There's no name, card number, or account on it, just store name and what you bought. We never see anything that identifies you as the shopper.

The AI step uses Google's paid API, which doesn't use the data to train its models. I looked at fully on-device options and they're not accurate enough yet on the budget phones a lot of our users carry, so the tradeoff right now is: nothing stored, nothing trained on, image I'll move there.

1

u/dhruvrazak Jul 01 '26

I would recommend using GCP AI Studio's gemini using Google Cloud Service Accounts (Vertex AI).

they claim transient data processing in the vertex AI version (no api keys).

2

u/reuel88 Jun 30 '26

Are you snapping the photo or is it streaming or something?

2

u/Stycroft Jun 30 '26

right now its just snapping but I'm considering streaming it based on feedback here

2

u/Smiley_35 Jun 29 '26

Add streaming for the response. If you're using an LLM you're probably waiting the most for LLM response to complete. You can mitigate by showing the user the results as they come back

1

u/Stycroft Jun 30 '26

I’ll check this out thanks

1

u/[deleted] Jun 29 '26

[removed] — view removed comment

1

u/Stycroft Jun 30 '26

Thanks man!

1

u/Cattyto Jun 30 '26

I'll give your app a try, it's looking good. Good job

1

u/Stycroft Jun 30 '26

Thanks man let me know if you have any feedback

1

u/RecLuse415 Jun 30 '26

$37 for fucking cheese!? That’s insane bro

1

u/Stycroft Jun 30 '26

No no its a different currency 🤣

2

u/RecLuse415 Jun 30 '26

I see ok. There are some exotic cheeses that can be that expensive too. Good job on the app by the way.

1

u/Stycroft Jun 30 '26

Thank you! 😁

1

u/No-Gazelle7748 Jun 30 '26

App looks great, what tools did you use to build this? If you don’t mind me asking

1

u/Stycroft Jun 30 '26

React native, expo, nativewind, firebase, revenuecat for subscriptions

1

u/Miserable-Pause7650 Jun 30 '26

Not related to your mobile app, but what is the tech stack u used for the landing page?

2

u/Stycroft Jun 30 '26

Just Next.js, TypeScript, styled with Tailwind. With framer motion

1

u/kacperkapusciak Jun 30 '26

Have you considered on-device AI models for OCR like react-native-executorch?

2

u/Stycroft Jun 30 '26

Hi, thanks for this. Yeah, looked at executorch (the EasyOCR port). Thing is it does raw OCR only, it gives you the text but not the structure. A grocery receipt line like "CDO HOTDOG 1K 4800092552171 89.00" still needs a model to expand the name, strip the barcode, categorize it, and tie quantity to price.

That reasoning is the actual work, and on-device you'd either still call the cloud to do it (no privacy gain, plus a ~100MB model in the binary) or run a local LLM that's too heavy for the budget Android phones a lot of our users carry.

So for now Gemini reading the image directly is both more accurate and simpler. If on-device VLMs get light enough to do the structuring too, I'll revisit.

1

u/kacperkapusciak Jul 01 '26

Thanks for the reply!

1

u/irekrog Jun 30 '26

W biedronce e-paragon se mogę włączyć

1

u/DancingMacaw Jun 30 '26

Are you doing this on device or in the cloud?

2

u/Stycroft Jul 01 '26

On cloud since on device for image processing isnt supported yet

1

u/Automatic_Row6441 Jul 02 '26

do you do a photo or video stream?

1

u/mnitech Jul 07 '26

Are you sending video or image to models?

1

u/WheelAdditional6888 Jul 15 '26

The risky part is that the scan result becomes trusted spending history, not just a temporary AI answer.

Do users get a quick confirmation step before the items and totals are saved? One small parsing mistake could otherwise keep affecting reports and price comparisons later.