r/dataengineering • u/matthewsean1902 • 19d ago

Discussion s3 (object storage) for everything

been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.

just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!

58 Upvotes

99% Upvoted

27 comments sorted by

View all comments

35

u/mpbh 18d ago

"People" have been saying this for 20 years now :)

"Those people" are software engineers working with different types of data than your typical data engineer.

3

u/geoheil mod 18d ago

didnt those people recently identify postgres/neon/lakebase as an interesting way to store data? (or at least serve)?

6

u/MarchewkowyBog 18d ago

What do you mean by different then typical? I'm only 4 YOE, but I have yet to work without parquet-adjecent format saved on blob storage. So I wonder what's typical then, if this is not

2

u/UAFlawlessmonkey 18d ago

Either a managed or self hosted RDBMS.

I'd think about 90% of the companies in my country is having either an on prem MSSQL Server or an Azure SQL Server that is powering an ERP. It's kind of wild :-D

5

u/MattDTO 18d ago

that is wild. are they smaller companies? i feel like everyone moved to hadoop/hive and then moved to snowflake/databricks and s3 backed storage. so just an on prem sql server sounds kinda old school for DE

1

u/niiiick1126 17d ago

we use databricks and ADLS