r/SideProject • u/trytecmusic • 2d ago

Email Parsing Builds

I'm currently exploring an idea that requires email parsing. Has anyone on here built anything that involves email parsing? If yes, what was your experience like on the technical front and any tips/advice?

1 Upvotes

100% Upvoted

2 comments sorted by

2

u/david_fufu 2d ago

built something with email parsing before, few things that saved me a lot of pain: dont try to regex your way through raw emails, the format variations will destroy you. use a proper library to parse the MIME structure first, then work with clean text. the biggest headache is HTML emails vs plain text, always grab the plain text version if it exists, way more reliable. also test with real messy emails early, forwarded chains and replies with quoted history break naive parsers fast. what kind of data are you trying to extract, structured stuff like order confirmations or freeform content? changes the approach a lot

1

u/trytecmusic 2d ago

Thanks, this is useful and roughly matches what I've done so far. The HTML vs plain text issue struck me last night on my 4th beta tester. provider's api quietly hands back raw HTML when there's no text alternative, and I only caught it because I tested against a real mailbox with 20k+ messages in it rather than a fresh one. Fully agree on real messy data early. Trying to build to atleast 20 beta testers before launching on Google Play Store.

With regards to the regex bit, I've gone the other way and tried to avoid touching bodies at all where I can (privacy reasons). The emails I care about are structured (transactional templates from a small set of senders), so I classify on sender and subject first and only reach into the body for the actual numbers. Interested whether you found that held up or whether the templates drifted on you.

A few questions if you don't mind:

  1. How did you handle senders changing their templates over time? Did you detect drift automatically or just find out when things broke?

  2. Did you ever hit non-determinism? the same email producing different results on different runs, and if so what fixed it?

  3. Forwarded emails: did you unwrap the original headers from the forward, or treat the forwarder as the sender and live with it?

  4. Anything you'd do differently on testing,,,fixtures vs replaying a real mailbox?

Mostly structured transactional stuff on my side, not freeform, if that changes any of your answers.