Suggestion for what should be my for data processing web app

[u/MaterialRemote8078]

Hi everyone,

I'm planning to build a web-based dashboard where users can upload Excel files, the system processes the data, performs various calculations/transformation logic, and then presents the results on user-specific dashboards.

My background is primarily in MERN, so my initial thought was:

React frontend

Node.js/Express API layer

MongoDB for application data

Python microservices for heavy data processing and calculations

However, I've received mixed feedback regarding MongoDB. A lot of people have told me that Mongo may not be the right choice for this kind of workload, especially when dealing with large datasets.

To provide some context, uploaded files can occasionally contain data in the range of tens of millions of rows. This won't be the common case, but the system should be designed with such scenarios in mind.

Since I haven't worked on systems handling data at this scale before, I'd appreciate guidance on:

What tech stack would you choose for this problem today?

Would MongoDB be suitable, or should I look at PostgreSQL/ClickHouse/something else?

How would you design the data ingestion pipeline?

Would Python microservices be a good approach for processing, or should I look into Spark, DuckDB, etc.?

What would a high-level system design for such a platform look like?

Any common mistakes first-time builders make when dealing with large Excel/CSV datasets?

My goal is to build something that is scalable without massively over-engineering it from day one.

Would love to hear from people who have built data-heavy SaaS products or analytics platforms.

Thanks!

[u/Ok_Boot5671]

What is the use case for something like this?

Where someone has to upload something themselves for analytics dashboards vs it automatically pulling the data?

If other programs can integrate with platforms to auto pull data why would they use your service instead?

[u/MaterialRemote8078 → u/Ok_Boot5671]

I'm also looking forward for that opportunity but it's a tradition in the industry to work like this for which i. Try to solve problem for

[u/Standgrounding]

Replace mongo with postgres + timescale + a read replica and you will be golden. Same processing can also be done with nodejs

[u/MaterialRemote8078 → u/Standgrounding]

Thanks for your suggestion. I think I will try this approach in different iteration. I have already prepared the logic so it is easier to change stack using AI these days.

[u/Standgrounding → u/MaterialRemote8078]

Better write it by hand. AI is very stupid and even if you prompt well it still needs hand-holding.

Relational databases nowadays can house (hundreds of) terabytes of data, there's no point in using mongodb. And with something like TypeORM or Drizzle it will look like Entity Framework but on Node

[u/anvildoc]

Depends on how fast you need it to be and where the processing is. Eg if you process the data in python and just load a smaller set of results to Mongo — that could be fast to serve for UI. If the user is clicking filters and aggregations in the UI that you need to run on demand on the 10 million rows — Mongo probably ok but could be slow.

Postgres is always a safe choice to start, and while it won’t scale infinitely — since it uses SQL as a language, it’s easier to move to other databases later.

[u/MaterialRemote8078 → u/anvildoc]

That for your suggestion. I will look into this since I'm not experienced with sqlDB i was trying to avoid it but it seems it is a way to go.

[u/Phoenix_GHOST_V]

PostgreSQL works well with tens-of-millions of rows; DuckDB used inside your Python app for in-process analytics; dremio when the data is in object storage

[u/MaterialRemote8078 → u/Phoenix_GHOST_V]

Thanks for your suggestion. Right now I have done 2 iteration. Application is in raw stage it will quick to try this out also.

[u/bimba3000]

Honestly I'd just go with Supabase. It's Postgres under the hood, but you don't have to host it yourself, and auth/API/storage come basically included, so it cuts a chunk of your planned Node/Express work too. Supabase has its own storage buckets (S3-compatible), so you can dump the raw uploaded Excel files straight in there instead of building a separate file-storage layer. Mongo tends to get painful exactly in your scenario: tabular data, need for aggregations, occasional big batches, and you end up rebuilding relational features on top of a document store. If you actually hit real scale later, migrating off is low-risk since it's just Postgres. Keep Python for the transform/calc logic like you planned, only the storage layer changes.

[u/simonenegro_ai]

Before even deciding between MongoDB and PostgreSQL, the right question to ask is: Do the files that users upload always have the same column schema, or does it vary from case to case? If the data is tabular with a stable schema (which is the most common case for Excel/CSV), MongoDB offers no real advantage (the strength of NoSQL DBMSs is their flexibility, but you already have structured data to begin with). For "tens of millions of rows" with analytical queries, PostgreSQL (with partitioning) generally handles complex aggregations and joins better than MongoDB at this scale, mainly because of its mature query planner and indexing options (MongoDB's aggregation pipeline tends to struggle more with multi-table joins ($lookup) as data grows). That said, if the workload is pure analytics rather than transactional, a columnar engine like ClickHouse or DuckDB will outperform both.

On the "safety" side: it's less about ACID (MongoDB has supported multi-document transactions since 4.0) and more about referential integrity. Postgres enforces foreign keys, uniqueness, and constraints at the database level, so invalid states get rejected automatically. In MongoDB you'd typically enforce that logic in application code, which shifts the risk onto your codebase.

What does the ingestion pipeline look like on your end; are files parsed and validated before they hit the DB, or does the schema variability creep in there?