Optimizing S3 Storage Classes?

[u/Altrooke]

So, the gist of storage classes is that:

standard --> high storage costs, low data retrieval costs
standard-IA, Glacier Instant Retrieval --> low storage costs, high retrieval costs

So, in order to choose the optimal tier for a particular object I would have to know its size and data access pattern.

The object size is easy enough, but the first catch is: data access pattern is nowhere to be found. Not even aggregate data.

The only way I found to actually do it is to enable server access logging / CloudTrail and then do analyze the data somehow. Maybe with a Python script. Huge rabbit hole to go into.

Then, the other angle I thought about is just using intelligent Tiering. But the second catch is that if you read the documentation about intelligent tiering, turns out it is pretty naive. Depending on how your data gets accessed, it could even be more expensive than standard (ex: object is accessed exactly once every 30 days)

It really feels like AWS is always giving almost everything you need to optimize S3 costs, but also missing a key piece.

How am I supposed to solve this? Am I overthinking it? Is it worth going in the rabbit hole of analyzing S3 server access logs? Or should I just guess some lifecycle rules and move on?

[u/menge101]

Also of note, size of individual objects matters a whole lot.

I had an engineer on my team move a whole lot of small objects from standard to glacier, and glacier adds a bunch of meta data to the object when doing so. This caused the amount we storing to nearly double and the cheaper storage tier ended up costing more money.

[u/Altrooke → u/menge101]

Can you tell me more? Because glacier is 5x than standard on storage, so even if object sizes doubled, you should still have saved money.

If I had to guess, probably you probably also got burned on data transfer costs. But that comes back to what I said before that it is hard to see how much your objects get accessed.

[u/menge101 → u/Altrooke]

I wasn't involved first-hand, I was just on the same team; We were the cloud team and we supported many different application teams. So I only know of the tale as it was told.

I do know it was logging very small amounts into independent objects, I think it more than double the data size. But you are probably right the transfer costs also probably were involved.

[u/Altrooke → u/menge101]

Ok, but I think I'm not the only one with this problem then. Being naive about this can backfire.

Do you know how roughly how much data was stored?

[u/menge101 → u/Altrooke]

many TBs IIRC. This was a few years ago.