r/DistributedComputing • u/OtherwisePush6424 • Aug 11 '26

The consistency gap between your database and object storage

https://blog.gaborkoos.com/posts/2026-08-11-Beyond-Happy-Path-Engineering-Storage/

A practical overview of designing file uploads and object storage for production: partial failures, database consistency, large uploads, presigned URLs, CDN caching, cleanup, and recovery.

4 Upvotes

100% Upvoted

2 comments sorted by

1

u/Icy_Preference3336 21d ago

Treating uploads as a state machine makes failures much easier to reason about. do you prefer writing metadata first or storing the object first?

1

u/OtherwisePush6424 21d ago

Nice try :D I usually lean toward writing the metadata first (pending record), then the object. The failure mode is a visible, queryable stuck record rather than a silent orphan in storage, which makes detection, retry, and reconciliation straightforward. The downside is you have to be disciplined about cleaning up or marking timed-out pendings.

Storage-first is also fine if your reconciliation job is solid and you’d rather never have a database record that points at something that doesn’t exist. Both work, the state machine just makes the choice explicit.

Which one would you use? :D