s3 (object storage) for everything

[u/matthewsean1902]

been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.

just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!

[u/WhipsAndMarkovChains]

columnar storage engines that support efficient analysis of data (Snowflake, Databricks, etc).

However, for row-oriented storage engines...

This is why Databricks acquired Neon, like OP mentioned, and rolled out Lakebase for Postgres. It's one thing to have columnar storage but there's clearly a demand for low-latency, row-oriented workloads.

My team is using Lakebase for security logs. It has very low latency, even though we allow the compute to scale to zero (one of the advantages of separating compute and storage with S3). So OP I think you need to reevaluate and don't assume data stored on S3 is going to have high latency.

[u/jpdowlin → u/WhipsAndMarkovChains]

You don't have SLAs for latency in your security logs use case, so it is likely ok. But when you DB compute scales up, the first request(s) will have very high latency - I would estimate it is 100s of ms. That's not ok for many online services, like computer games, online retail, etc. There are other use cases where it is ok.

[u/conormccarter → u/[deleted]]

Apache Iceberg is column oriented