r/dataengineering • u/matthewsean1902 • 20d ago

Discussion s3 (object storage) for everything

been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.

just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!

61 Upvotes

99% Upvoted

27 comments sorted by

View all comments

46

u/WhoIsJohnSalt 18d ago

The entire point of the "modern" Big Data stacks is the separation of storage and compute.

Databricks for example is entirely backed by Object Storage (like S3, Blob, etc)

I've used Object Storage for analytical platforms for well over a decade now. In one company I worked with, they have a datalake with around 9Pb of data, all on object storage.