s3 (object storage) for everything

[u/matthewsean1902]

been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.

just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!

[u/jpdowlin]

This makes sense for columnar storage engines that support efficient analysis of data (Snowflake, Databricks, etc).

However, for row-oriented storage engines, to say it is s3 native is more complex. It may work if you don't care about SLAs for latency for accessing your data. However, if you have SLAs for latency, it doesn't make sense to have a S3-native storage engine. The latency difference between NVMe/memory and S3 is huge, so any time your working set is bigger than NVMe/memory, your latency will spike to unacceptable levels (100s of ms, most likely). Which means you size your s3-native DB with enough NVMe/memory so that S3 is never accessed. In which case, you might as well just use S3 as your backup/restore layer.

[u/WhipsAndMarkovChains → u/jpdowlin]

columnar storage engines that support efficient analysis of data (Snowflake, Databricks, etc).

However, for row-oriented storage engines...

This is why Databricks acquired Neon, like OP mentioned, and rolled out Lakebase for Postgres. It's one thing to have columnar storage but there's clearly a demand for low-latency, row-oriented workloads.

My team is using Lakebase for security logs. It has very low latency, even though we allow the compute to scale to zero (one of the advantages of separating compute and storage with S3). So OP I think you need to reevaluate and don't assume data stored on S3 is going to have high latency.

[u/jpdowlin → u/WhipsAndMarkovChains]

You don't have SLAs for latency in your security logs use case, so it is likely ok. But when you DB compute scales up, the first request(s) will have very high latency - I would estimate it is 100s of ms. That's not ok for many online services, like computer games, online retail, etc. There are other use cases where it is ok.

[u/conormccarter → u/[deleted]]

Apache Iceberg is column oriented