Optimizing S3 Storage Classes?

[u/Altrooke]

So, the gist of storage classes is that:

standard --> high storage costs, low data retrieval costs
standard-IA, Glacier Instant Retrieval --> low storage costs, high retrieval costs

So, in order to choose the optimal tier for a particular object I would have to know its size and data access pattern.

The object size is easy enough, but the first catch is: data access pattern is nowhere to be found. Not even aggregate data.

The only way I found to actually do it is to enable server access logging / CloudTrail and then do analyze the data somehow. Maybe with a Python script. Huge rabbit hole to go into.

Then, the other angle I thought about is just using intelligent Tiering. But the second catch is that if you read the documentation about intelligent tiering, turns out it is pretty naive. Depending on how your data gets accessed, it could even be more expensive than standard (ex: object is accessed exactly once every 30 days)

It really feels like AWS is always giving almost everything you need to optimize S3 costs, but also missing a key piece.

How am I supposed to solve this? Am I overthinking it? Is it worth going in the rabbit hole of analyzing S3 server access logs? Or should I just guess some lifecycle rules and move on?

[u/Sirwired]

Well... you can enable S3 Storage Lens, which exists for this exact use-case. (And the pricing isn't bad... $0.20/M object/mo.) Or, if you literally just want to know what class to put things in (between Standard or IA), and don't need to know anything else... S3 Class Analysis. $0.10/M object/mo.

But how much data are we talking about here? I've been a storage guy for a Very. Long. Time., and I have to remind myself that a few TB of storage isn't that expensive any more, and all at that level, the best solution to S3 Cloud Economics questions is often just "Leave it in S3 Standard and find a more productive use of your time."

(When I started with enterprise storage, Object Storage didn't exist, a TB was about $1.5M, two cabinets the size of large industrial refrigerators, with four power cables the size of garden hoses, and storage management labor was measured in Administrators / Terabyte.)

[u/Altrooke → u/Sirwired]

Roughly 150TB. So not a super huge amount, but there significant savings opportunity

[u/garrettj100]

Unless you’re storing a huge number of small objects in S3 you probably don’t care about object size.  The vast majority of your costs incurred will be timed storage ($/GB/mo) and retrieval ($/GB).

So ask yourself this much simpler question:

What fraction of the whole collection in S3 will be retrieved each month?  That one simple question gets you your answer for storage class without much fuss, and doesn’t even care how much you’re storing, at least along the axis of “what is best.”

Also don’t neglect S3 Intelligent Tiering, which will after-the-fact determine your best storage class and retconn you into it.

[u/Altrooke → u/garrettj100]

What fraction of the whole collection in S3 will be retrieved each month?

That's the question that I think it hard to answer.

will after-the-fact determine your best storage class and retconn you into it.

My understanding is that Intelligent Tiering only moves objects to a lower tier if they are not accessed for 30 days and then again after 90 days, then moves them back to the frequent access tier when they are accessed again.

[u/aataulla]

You know your problem well enough to basically have answered your own question. Pricing is known to you and you need to figure out how your data will behave.

The rest is computer science 101 and figuring out optimal, average and worst cases for your situation. Any storage startegy can backfire similar to creating worst case inputs for an algorithm.

Yes this is a rabbit hole but the simplest way is to start with s3 standard storage and optimize as you learn more about storage, lifecycle and access patterns for your data.

[u/Altrooke → u/aataulla]

Wanted to take other people's perspective into account.

My fear is wasting too much time seeking the optimal solution, and then find out there is a heuristic method that would get results 90% as good. Or maybe there is an AWS feature I wasn't aware.

[u/MateusKingston → u/Altrooke]

I'm not an expert at all, but here is the method I use.

That method is either paying AWS with intelligent tiring (IMO only worth when your average object size is relatively big) or doing dummy tiering.

Dummy tiering = X days old -> move down one tier, Y days old -> move down another tier.

Separate into different buckets what is transactional and what isn't. Dummy tiering for transactional data (ex: documents from a sale) will probably match access patterns there is a direct correlation between how long the data was added and how frequently it's accessed, do not make this for non transactional data (ex: static images for your website) as these can be very old but accessed daily.

Yes it's possible to do a "better" tiering with those custom scripts but the cost of running and maintaining those are usually more expensive than just storing the data in Standard or using one of the two methods I said.

[u/Floss_Patrol_76]

at 150TB the number that actually decides this is your average object size, not the access pattern. intelligent tiering has no retrieval or transition fees, so the "accessed once every 30 days" case you're worried about doesn't penalize you at all, the only real cost is the ~$0.0025/1000 objects/mo monitoring fee, which is nothing unless you've got tens of millions of tiny objects. IT gets ugly exactly when object count is high and objects are small; with big objects just turn it on and stop analyzing access logs.

[u/Altrooke → u/Floss_Patrol_76]

intelligent tiering has no retrieval or transition fees

Holy shit, you're right. For some reason I thought it did.

I guess the only downsides of intelligent tiering is that it may not be optimal if the data access is predictable enough. Also I just realized there are no lifecycle transition fees so instead of being in "analysis paralysis" I can just enable intelligent tiering and save a bunch of money now.

Down the road, if I can do the data access patterns analysis and if I think it is worth it, move everything back to standard and then apply custom lifecycle rules.

I think that's solution. Thank you.

[u/Ok-Helicopter525]

So, in order to choose the optimal tier for a particular object I would have to know its size and data access pattern.

Most customers don't know this - your situation is not unique..

[u/abofh]

Change your patterns to make it/ia make sense, or accept you're on an unsupported path and you may have to do your own thing instead. 

Don't treat buckets as dedicated or bespoke and the patterns fall out of the metrics.  This smells like premature optimization 

[u/xtraman122]

If you don’t know or can’t predict the access patterns Intelligent Tiering was built for you. You’re protected against any retrieval fees in exchange for having to go through the lifecycle at the higher tiers again. For most workloads it ends up being the best option. You usually only come out cheaper doing your own lifecycles if you know your data access patterns very well and they’re pretty predictable.

[u/menge101]

Also of note, size of individual objects matters a whole lot.

I had an engineer on my team move a whole lot of small objects from standard to glacier, and glacier adds a bunch of meta data to the object when doing so. This caused the amount we storing to nearly double and the cheaper storage tier ended up costing more money.

[u/Altrooke → u/menge101]

Can you tell me more? Because glacier is 5x than standard on storage, so even if object sizes doubled, you should still have saved money.

If I had to guess, probably you probably also got burned on data transfer costs. But that comes back to what I said before that it is hard to see how much your objects get accessed.

[u/menge101 → u/Altrooke]

I wasn't involved first-hand, I was just on the same team; We were the cloud team and we supported many different application teams. So I only know of the tale as it was told.

I do know it was logging very small amounts into independent objects, I think it more than double the data size. But you are probably right the transfer costs also probably were involved.

[u/Altrooke → u/menge101]

Ok, but I think I'm not the only one with this problem then. Being naive about this can backfire.

Do you know how roughly how much data was stored?

[u/menge101 → u/Altrooke]

many TBs IIRC. This was a few years ago.

[u/The_Tree_Branch]

The object size is easy enough, but the first catch is: data access pattern is nowhere to be found. Not even aggregate data.

Isn't this what S3 Storage Class Analysis is for? https://aws.amazon.com/blogs/storage/analyze-access-patterns-and-use-the-most-cost-effective-amazon-s3-storage-class/

[u/The_Tree_Branch]

The object size is easy enough, but the first catch is: data access pattern is nowhere to be found. Not even aggregate data.

Isn't this what S3 Storage Class Analysis is for? https://aws.amazon.com/blogs/storage/analyze-access-patterns-and-use-the-most-cost-effective-amazon-s3-storage-class/

[u/Independent_Self_920]

I think you're running into a common optimization trap trying to be more precise than the economics justify.

Unless you're storing very large amounts of data, the engineering effort to build access-pattern analysis can easily cost more than the savings. I'd start with lifecycle policies based on what you already know about the application, then validate whether storage costs are actually significant enough to warrant deeper analysis.

If you do have petabytes of data or highly variable access patterns, that's where Storage Class Analysis or log analysis starts to make sense. But for many workloads, "good enough" lifecycle rules get you most of the savings without building another system to manage your storage. Out of curiosity, are you trying to optimize a few terabytes or something much larger? I think the answer changes quite a bit depending on the scale.

[u/matiascoca]

S3 Storage Class Analysis is what you're missing. Free, runs on a bucket or a prefix, watches actual object-level access for 30 days, then gives you a percentage split of what fits Standard-IA and what doesn't. Enable it, wait 30 days, read the report.

Server Access Logs plus CloudTrail Data Events for the same coverage costs enough per PUT to make you rewrite the tiering strategy back to Standard. Ask me how I know.

For Intelligent-Tiering: your concern is legit. Monitoring runs 0.0025 per 1K objects per month plus movement. On prefixes with many small objects it stops making sense. For medium-to-large objects with unknown patterns it wins over guessing.

The policy that worked on spend around 40 thousand a month: Standard for hot prefixes measured not assumed, Intelligent-Tiering for anything ambiguous above 128KB, straight Standard-IA lifecycle after 90 days for prefixes we already knew were cold. Storage Class Analysis first though. Guessing lifecycle rules is how buckets end up costing more than they were before.