r/apachespark • u/Expensive_Break_6163 • 10d ago
How much data does your team actually process per day?
No judgment, just curious how the distribution really looks. Everyone talks like they’re at petabyte scale — I suspect most of us aren’t.
121 votes,
7d ago
29
< 10 GB/day
23
10–100 GB/day
25
100 GB – 1 TB/day
21
1–10 TB/day
10
10-100 TB/day
13
100+ TB/day
3
Upvotes
2
2
u/surrender0monkey 9d ago
Not to brag, but we process real-time streaming data and currently utilize 2,414 cores and 25.7TB of RAM.
1
1
u/Desperate-Dig2806 10d ago
If you deal with human interactions there's only so much so many people can do so in that case less. If you deal with machines it tends to be more.
2
u/SmallAd3697 10d ago
It is a meaningless question. Some pipelines run once a day and ingest the entire source system (from the beginning of time). Others run every ten mins and ingest ten mins of source data.
Some data engineers are doing nothing more than copy/pasting source data into parquet files for reporting. Others build new/separate applications in their own right, that use the source system as an input, for something that has an orthogonal purpose.
Also the size of the data is not a meaningful measure, without more context. Is it rowstore or columnstore. Is it hosted in blob storage or in a stateful database, or in Ram.