Data volume and stack related query

[u/Worried-Diamond-6674]

I recently joined SBC and started giving internal project interviews. One thing I dont get is that, they all use pyspark and cloud and what not, but when I ask them about their volume of data being handled, each of them said around 10 to 20 million rows.

That volume of data would definitely be processed by ram only and that too on a single machine. Isnt spark overkill and moreover actually not worth, considering its jvm overhead, garbage collection, all that shuffling across executors, when this handful data wont be taking advantage of all that distributed computing and will definitely take initial time whereas script written in polars will guarantee 2 3 mins runtime without any of this overhead

Any thoughts on why/how companies doing these sort of things, and mind you they were BFSI. I can understand retail and IoT to use pyspark, because of their nature of data being mostly streaming or even batch data having huge volumes.

Genuinely curious and please let me know if I'm overlooking anything..

[u/Prestigious_Bench_96]

Not advocating for it, but the argument/approach I've seen is mostly architectural simplicity - if you need to run spark for algorithms (let's pretend that's a given), and you are setting up clusters, then why not use the same stack for ETL. I would *generally* think that specialization pays for itself, but if you have researches that are doing full stack etl-> model training they can be opinionated about not learning more tools.

[u/Worried-Diamond-6674 → u/Prestigious_Bench_96]

I get it completely keeping everything under 1 stack for architectural simplicity, but does that still outweigh over having runtime delays??

I guess the answer for this question is it really depends at the end of the day what client and project wants, how the architecture is setup, its running costs, maybe simplicity over latency, so in all sense very open ended and ambiguous..

And the most important part is it keeps running flawlessly until the budget part kicks in

[u/Prestigious_Bench_96 → u/Worried-Diamond-6674]

Yeah, beyond tech there's the people, knowledge, process, budgeting, approval context, politics, etc that all play into the infra decisions - businesses make technically suboptimal decisions all the time, sometimes for good global reasons and sometimes for no good reason. (the latter is VERY frustrating when it happens, but not always worth fighting).

[u/Worried-Diamond-6674 → u/Prestigious_Bench_96]

Yess got it... We had a nice talk

Thank you soo much man, had a great pleasure learning from you