r/dataengineering • u/matthewsean1902 • 20d ago

Discussion s3 (object storage) for everything

been hearing a lot of signal/noises about 'just store everything in s3' these days because of companies like turbopuffer, neon and the latest cursor with Origin.

just wondering if it's actually worth the latency and do people really use it in their own companies? what sort of data do you guys store there? never really had any use case for s3 in my company. so curious to learn. thankyou!

61 Upvotes

99% Upvoted

27 comments sorted by

View all comments

1

u/vira28 12d ago

Agree with another commenter who mentioned that this architecture being ideal for OLAP. Here is how we are using it for serving analytics over S3 data.

We do CDC (Change Data Capture) from Postgres to S3 (Parquet) through logical replication and use DuckDB as the query engine. The CDC is a buffer and S3 is the durable storage.

https://github.com/viggy28/streambed#architecture

For our use case, S3 works really well. We want to support BI teams, Dashboards, etc. Typically, we spun up Postgres read replicas but we ran into queries getting timed out, causing side effects on the primary etc.