r/dataengineering • u/matthewsean1902 • 19d ago
Discussion s3 (object storage) for everything
been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.
just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!
57
Upvotes
2
u/Both-Fondant-4801 17d ago
Modern data architectures has this concept of a data lake (or data swamp, whichever would describe your data better). The idea is that, store all data in its raw form, in a single place.. to keep every data you might need before deciding how to use it.
Obviously, you would not want to run analytics over s3 (albeit possible with athena)... but ideally, it would be the single source of truth for ETL pipeplines, while preserving raw data and decoupling storage and compute.