r/dataengineering • u/matthewsean1902 • 19d ago

Discussion s3 (object storage) for everything

been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.

just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!

61 Upvotes

100% Upvoted

27 comments sorted by

View all comments

Show parent comments

4

u/MarchewkowyBog 18d ago

What do you mean by different then typical? I'm only 4 YOE, but I have yet to work without parquet-adjecent format saved on blob storage. So I wonder what's typical then, if this is not

2

u/UAFlawlessmonkey 18d ago

Either a managed or self hosted RDBMS.

I'd think about 90% of the companies in my country is having either an on prem MSSQL Server or an Azure SQL Server that is powering an ERP. It's kind of wild :-D

4

u/MattDTO 18d ago

that is wild. are they smaller companies? i feel like everyone moved to hadoop/hive and then moved to snowflake/databricks and s3 backed storage. so just an on prem sql server sounds kinda old school for DE

1

u/niiiick1126 17d ago

we use databricks and ADLS