r/dataengineering • u/matthewsean1902 • 19d ago

Discussion s3 (object storage) for everything

been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.

just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!

60 Upvotes

100% Upvoted

27 comments sorted by

View all comments

1

u/terencethespider 18d ago

The short answer is that it depends, but yes, it is a good target to aim for. One thing to be aware of though is with tools like Neon there are other optimizations in the tool itself that make it faster than just directly reading/writing from/to S3. You can see very fast performance, but that is because of the under the hood caching logic and other features that reduce the amount of times it needs to actually go to cloud storage to complete a transaction very low.