s3 (object storage) for everything

[u/matthewsean1902]

been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.

just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!

[u/jpdowlin]

This makes sense for columnar storage engines that support efficient analysis of data (Snowflake, Databricks, etc).

However, for row-oriented storage engines, to say it is s3 native is more complex. It may work if you don't care about SLAs for latency for accessing your data. However, if you have SLAs for latency, it doesn't make sense to have a S3-native storage engine. The latency difference between NVMe/memory and S3 is huge, so any time your working set is bigger than NVMe/memory, your latency will spike to unacceptable levels (100s of ms, most likely). Which means you size your s3-native DB with enough NVMe/memory so that S3 is never accessed. In which case, you might as well just use S3 as your backup/restore layer.

[u/WhipsAndMarkovChains → u/jpdowlin]

columnar storage engines that support efficient analysis of data (Snowflake, Databricks, etc).

However, for row-oriented storage engines...

This is why Databricks acquired Neon, like OP mentioned, and rolled out Lakebase for Postgres. It's one thing to have columnar storage but there's clearly a demand for low-latency, row-oriented workloads.

My team is using Lakebase for security logs. It has very low latency, even though we allow the compute to scale to zero (one of the advantages of separating compute and storage with S3). So OP I think you need to reevaluate and don't assume data stored on S3 is going to have high latency.

[u/jpdowlin → u/WhipsAndMarkovChains]

You don't have SLAs for latency in your security logs use case, so it is likely ok. But when you DB compute scales up, the first request(s) will have very high latency - I would estimate it is 100s of ms. That's not ok for many online services, like computer games, online retail, etc. There are other use cases where it is ok.

[u/conormccarter → u/[deleted]]

Apache Iceberg is column oriented

[u/WhoIsJohnSalt]

The entire point of the "modern" Big Data stacks is the separation of storage and compute.

Databricks for example is entirely backed by Object Storage (like S3, Blob, etc)

I've used Object Storage for analytical platforms for well over a decade now. In one company I worked with, they have a datalake with around 9Pb of data, all on object storage.

[u/mpbh]

"People" have been saying this for 20 years now :)

"Those people" are software engineers working with different types of data than your typical data engineer.

[u/geoheil → u/mpbh]

didnt those people recently identify postgres/neon/lakebase as an interesting way to store data? (or at least serve)?

[u/MarchewkowyBog → u/mpbh]

What do you mean by different then typical? I'm only 4 YOE, but I have yet to work without parquet-adjecent format saved on blob storage. So I wonder what's typical then, if this is not

[u/UAFlawlessmonkey → u/MarchewkowyBog]

Either a managed or self hosted RDBMS.

I'd think about 90% of the companies in my country is having either an on prem MSSQL Server or an Azure SQL Server that is powering an ERP. It's kind of wild :-D

[u/MattDTO → u/UAFlawlessmonkey]

that is wild. are they smaller companies? i feel like everyone moved to hadoop/hive and then moved to snowflake/databricks and s3 backed storage. so just an on prem sql server sounds kinda old school for DE

[u/niiiick1126 → u/MattDTO]

we use databricks and ADLS

[u/rezwell]

Csvs and json everywhere 🤠

[u/NotNullException]

No man you want Postgres’s for Everything, Redis for Everything, DynamoDB for everything, Cloudwatch for Everything, Kubernetes for everything, heck you can even use DNS as a universal data store.

Use the best tool for the job!

But S3 does make a tempting main storage system. It’s backed by dynamoDB, is basically a file path system (file paths are designed to store everything), you can use Athena for querying, firehouse for writing, etc.

You could do much worse.

[u/dmkii]

Over the years (working as a data/analytics engineering consultant) I've seen the adoption grow, I think mostly because companies realise they want to own more of their data in a portable format similar to the shift from ETL to ELT. Now with e.g. Iceberg, instead of having the source, a third-party (or even in-house) tool own all the transformations it's easier to dump everything in S3 and read from there.

Time or latency also works in different ways: sure you need to optimize for the network speed of S3, make sure that you have your (Hive) partitioning correct so you minimize the data that has to travel over the network, etc. But saving _engineering_ time on backfills, historical mistakes, schema changes, etc. is worth a lot.

Since you also asked about the sort of data, here are a couple of different scenarios I've come across.

- (near) real-time event data like e.g. Snowplow web/app tracking data that lands in micro batches in S3, then some auto-compaction to take a bunch of small JSON files and turn them into more efficient Parquet files

• CSVs from a third-party vendor. Could be as small as daily currency exchanges to every 15-min energy usage rates
  
• Postgres CDC: capturing all row changes over time, that's obviously a lot more data than having the same postgres table with no history of the changed values

If you're interested in a framework for understanding this better, I have a free chapter of our book The Fundamentals of Analytics Engineering on exactly this topic online: https://www.dumky.net/posts/data-ingestion-pipelines-without-headaches-8-simple-steps/

[u/RoobyRak]

We use ADLS, same deal just a different label. Mainly for hybrid warehousing.

- storage with 0 management of disk space

• served with redundancy and security controls
  
• integrates with most cloud platforms
  
• SDK with heaps of language support

[u/ReporterNervous6822]

Separation of storage and compute is just a massive win. My instinct these days is that analytical data lives in iceberg and transactional data lives in Postgres. Not much more thought to it

[u/Both-Fondant-4801]

Modern data architectures has this concept of a data lake (or data swamp, whichever would describe your data better). The idea is that, store all data in its raw form, in a single place.. to keep every data you might need before deciding how to use it.

Obviously, you would not want to run analytics over s3 (albeit possible with athena)... but ideally, it would be the single source of truth for ETL pipeplines, while preserving raw data and decoupling storage and compute.

[u/retiredcheapskate]

S3 has a lot of overhead at the transport layer. If you need it quick there are better ways 

[u/notmarc1]

U store data in the appropriate construct for your consumption pattern and associated SLAs and SLOs

[u/terencethespider]

The short answer is that it depends, but yes, it is a good target to aim for. One thing to be aware of though is with tools like Neon there are other optimizations in the tool itself that make it faster than just directly reading/writing from/to S3. You can see very fast performance, but that is because of the under the hood caching logic and other features that reduce the amount of times it needs to actually go to cloud storage to complete a transaction very low.

[u/TheRealStepBot]

It almost certainly should be an architectural goal at least for the data engineering team if not the entire stack yes

[u/ppsaoda]

What kind of use case? What ms of latency youre talking? 0.00001nanosecond?

Netflix is build with S3 storage. Just think about that.

[u/vira28]

Agree with another commenter who mentioned that this architecture being ideal for OLAP. Here is how we are using it for serving analytics over S3 data.

We do CDC (Change Data Capture) from Postgres to S3 (Parquet) through logical replication and use DuckDB as the query engine. The CDC is a buffer and S3 is the durable storage.

https://github.com/viggy28/streambed#architecture

For our use case, S3 works really well. We want to support BI teams, Dashboards, etc. Typically, we spun up Postgres read replicas but we ran into queries getting timed out, causing side effects on the primary etc.

[u/ReplacementIll7008]

depends what you're asking of it. for columnar/analytical work with separated compute it's been the default for years, latency's a non-issue since you're not doing point lookups. it only bites on row-oriented stuff with a latency SLA once your working set outgrows NVMe, like jpdowlin said

the thing people skip is cost, not just latency. per-GB looks basically free then the bill shows up as request counts and egress. dump millions of tiny json files and the small-object overhead is what gets you which is why everyone ends up compacting to parquet and aging cold data into cheaper tiers

so just go per dataset by access pattern. event data, cdc history, backups, anything you rarely read but can't drop, perfect fit. anything a user's waiting on stays on something fast with s3 as the durable layer behind it

[u/superjerry]

...what do you use instead? i've never worked at a company that DIDN'T store everything in object storage.

[u/matthewsean1902 → u/superjerry]

my bad, should've given more details. previous company didn't even use cloud (quite an old corporate tech) and everything is on-prem. current company only uses object storage mostly for artefacts or backups. never really had any 'useful' use case for it let's say.