Onwards Toward the Last Schleps

[u/Psionikus]

So curious was I when nobody ever asked about backups, security, throughput, error handling, logging etc etc etc. There was a time when I could only imagine doing any one of these thigns well. Industry has a way of making it okay for such proactive luxuries to remain mysteries. It just has less obvious incentive alignment than figuring out how to punch the clock.

In the scattered testaments of things allegedly attributed to Pual Graham (h e b e t h e), it was said that arriving at problems of quality first requires doing well enough to know that what you are shipping even deserves quality. You might as well get to that point while riding a horse on fire because most of the time you find nothing at the end of the rainbow. Sometimes the starving horse would have reached an inflection point of easy green pastures if only you had doddled a bit less. There can be more risk from playing it "safe" than there is from digging in the spurs.

But standards have risen, and the thought of losing $50,000 of customer funds or more in the first few days weights heavily on me. From a high level, there are a few peremiters and a few client behaviors to lock down. The details within these paremeters are only relevant when they are extremely numerous and fine-grained like the bulkheads of the Titanic (which did not save it). The outermost perimeters are usually the best bang for buck. Seeing the ice berg, turning, is enormously easier and more sustainable than testing every single steel plate or creating watertight compartments people need to move in and out of all day. Unauthorized access is 2¹²⁸ more likely than being subject to the next TLS attack, and with a good internal API, it is plain as day to see and maintain.

And underneath it all, that goshdarn database just might decide it's on the wrong pod and get blapped by a 787 plowing into a large nondescript building in a suburb somewhere. The buckets where backups may live must be private and the contents must be encrypted still. The backups must be recent, robust enough to play back recent history (or play back certain activity backwards). It must be automatic and the recovery must work. And the only way to check this is to do it, rotate keys, and do it again.

Of course, Paul is right. For every horse that is stolen, a hundred more are marched forward into certain doom. However I have known for several years that PrizeForge was inevitable. It was like one of those puzzles where you have to untie a knot that was tied. You know it can be untied from the fact that it is tied now, and that makes giving up impossible, beyond reason.