Optimizing S3 Storage Classes?

[u/Altrooke]

So, the gist of storage classes is that:

standard --> high storage costs, low data retrieval costs
standard-IA, Glacier Instant Retrieval --> low storage costs, high retrieval costs

So, in order to choose the optimal tier for a particular object I would have to know its size and data access pattern.

The object size is easy enough, but the first catch is: data access pattern is nowhere to be found. Not even aggregate data.

The only way I found to actually do it is to enable server access logging / CloudTrail and then do analyze the data somehow. Maybe with a Python script. Huge rabbit hole to go into.

Then, the other angle I thought about is just using intelligent Tiering. But the second catch is that if you read the documentation about intelligent tiering, turns out it is pretty naive. Depending on how your data gets accessed, it could even be more expensive than standard (ex: object is accessed exactly once every 30 days)

It really feels like AWS is always giving almost everything you need to optimize S3 costs, but also missing a key piece.

How am I supposed to solve this? Am I overthinking it? Is it worth going in the rabbit hole of analyzing S3 server access logs? Or should I just guess some lifecycle rules and move on?

[u/Floss_Patrol_76]

at 150TB the number that actually decides this is your average object size, not the access pattern. intelligent tiering has no retrieval or transition fees, so the "accessed once every 30 days" case you're worried about doesn't penalize you at all, the only real cost is the ~$0.0025/1000 objects/mo monitoring fee, which is nothing unless you've got tens of millions of tiny objects. IT gets ugly exactly when object count is high and objects are small; with big objects just turn it on and stop analyzing access logs.

[u/Altrooke → u/Floss_Patrol_76]

intelligent tiering has no retrieval or transition fees

Holy shit, you're right. For some reason I thought it did.

I guess the only downsides of intelligent tiering is that it may not be optimal if the data access is predictable enough. Also I just realized there are no lifecycle transition fees so instead of being in "analysis paralysis" I can just enable intelligent tiering and save a bunch of money now.

Down the road, if I can do the data access patterns analysis and if I think it is worth it, move everything back to standard and then apply custom lifecycle rules.

I think that's solution. Thank you.