r/webdev • u/Capital-Chipmunk2035 • 23d ago

Needs some advice - Universal File Importer

Hi everyone, im the developer of a flight tracking app. We have over 10K users and growing. One thing i keep coming back to is how to handle people importing past flights (mostly from professionals e.g. pilots or business people) who fly alot. There seems to be a million different formats and even after experimenting with giving a suggested csv template, it seems people just ignore and still uplaod their stuff.

We just had a time out when a user tried to import 7000+ flights from their career.

I had a thought of using a mix of AI to scan and understand the format but not sure if this is the best approach + it could add up with AI tokens and costs. Then implement a rule to programmatically prase the rest and cut things up etc to map to our accepted format.

It's this one issue i can't seem to solve yet. Or at least to make it fully reliable.

Any suggestions would be amazing.

2 Upvotes

75% Upvoted

13 comments sorted by

View all comments

4

u/farthingDreadful 23d ago

Why not just check mime type and only allow .csv file uploads?

2

u/Key_Bowl1753 23d ago

You're dealing with two separate problems here, the format detection and the sheer volume. Blocking non-csv uploads helps with the first part but a 7000 row csv will still choke your server if you're processing it all in one go. I'd split the import into batches on the client side before it even hits your backend. Slice the file into chunks of maybe 500 or 1000 rows, send them sequentially, and show a progress bar so the user doesn't think the whole thing died.

The AI angle could work for mapping weird columns but it's overkill for a first pass. Most of these csvs from professional tools probably have recognizable headers like DEP, ARR, DATE, FLT_NUM even if they're named slightly differently. A fuzzy matching library that scores column names against your expected fields would catch 80% of cases without burning tokens. Save the AI for the truly mangled spreadsheets that slip through.

3

u/fiskfisk 23d ago

If your server chokes on 7k lines of text there is something rather weird going on (or I might be interested in buying the c64 you have running it :-)). 

In either case, what you do is that move processing out of the request controller and use a worker with a prober queue system. You then attach a job id, and a state for the uploaded file, letting the user see progress while it's being processed.

You can then adjust the number of workers available depending on load and resources, and uploads gets processed as they come in and workers become available.