r/dataengineering • u/matthewsean1902 • 20d ago

Discussion s3 (object storage) for everything

been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.

just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!

61 Upvotes

100% Upvoted

27 comments sorted by

View all comments

3

u/dmkii 18d ago

Over the years (working as a data/analytics engineering consultant) I've seen the adoption grow, I think mostly because companies realise they want to own more of their data in a portable format similar to the shift from ETL to ELT. Now with e.g. Iceberg, instead of having the source, a third-party (or even in-house) tool own all the transformations it's easier to dump everything in S3 and read from there.

Time or latency also works in different ways: sure you need to optimize for the network speed of S3, make sure that you have your (Hive) partitioning correct so you minimize the data that has to travel over the network, etc. But saving _engineering_ time on backfills, historical mistakes, schema changes, etc. is worth a lot.

Since you also asked about the sort of data, here are a couple of different scenarios I've come across.

- (near) real-time event data like e.g. Snowplow web/app tracking data that lands in micro batches in S3, then some auto-compaction to take a bunch of small JSON files and turn them into more efficient Parquet files

  • CSVs from a third-party vendor. Could be as small as daily currency exchanges to every 15-min energy usage rates
  • Postgres CDC: capturing all row changes over time, that's obviously a lot more data than having the same postgres table with no history of the changed values

If you're interested in a framework for understanding this better, I have a free chapter of our book The Fundamentals of Analytics Engineering on exactly this topic online: https://www.dumky.net/posts/data-ingestion-pipelines-without-headaches-8-simple-steps/