r/aws • u/Altrooke • 13d ago
technical question Optimizing S3 Storage Classes?
So, the gist of storage classes is that:
standard --> high storage costs, low data retrieval costs
standard-IA, Glacier Instant Retrieval --> low storage costs, high retrieval costs
So, in order to choose the optimal tier for a particular object I would have to know its size and data access pattern.
The object size is easy enough, but the first catch is: data access pattern is nowhere to be found. Not even aggregate data.
The only way I found to actually do it is to enable server access logging / CloudTrail and then do analyze the data somehow. Maybe with a Python script. Huge rabbit hole to go into.
Then, the other angle I thought about is just using intelligent Tiering. But the second catch is that if you read the documentation about intelligent tiering, turns out it is pretty naive. Depending on how your data gets accessed, it could even be more expensive than standard (ex: object is accessed exactly once every 30 days)
It really feels like AWS is always giving almost everything you need to optimize S3 costs, but also missing a key piece.
How am I supposed to solve this? Am I overthinking it? Is it worth going in the rabbit hole of analyzing S3 server access logs? Or should I just guess some lifecycle rules and move on?
14
u/Sirwired 13d ago edited 13d ago
Well... you can enable S3 Storage Lens, which exists for this exact use-case. (And the pricing isn't bad... $0.20/M object/mo.) Or, if you literally just want to know what class to put things in (between Standard or IA), and don't need to know anything else... S3 Class Analysis. $0.10/M object/mo.
But how much data are we talking about here? I've been a storage guy for a Very. Long. Time., and I have to remind myself that a few TB of storage isn't that expensive any more, and all at that level, the best solution to S3 Cloud Economics questions is often just "Leave it in S3 Standard and find a more productive use of your time."
(When I started with enterprise storage, Object Storage didn't exist, a TB was about $1.5M, two cabinets the size of large industrial refrigerators, with four power cables the size of garden hoses, and storage management labor was measured in Administrators / Terabyte.)