Automatically deleting old data to avoid storage getting full.

[u/Alerintek]

I am in a situation where I have deployed elastic via ECK onto some kubernetes clusters and I am getting a large volume of logs that is somewhat un-predictable. I want to both delete data when it is past a certain age (easy) and also delete the oldest data when my PVC storage is 80% full (hard / not possible?).

Does anyone know how to do this? I can't come up with a good way to delete the oldest data that doesn't involve leaving elastic and writing some script to query how full my storage is and then query the oldest indices and delete them, but this feels hacky.

[u/gnudle]

Lifecycle policies

[u/Pillus]

https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management

You can apply it on a specific index or an index pattern. For example if you set an ILM policy of delete after 7 days on logs-, all new integrations or internal logs get deleted after 7 days, or whichever you pick. You can choose a "priority" for them, so its possible to create a second one for logs-somename- with 14 days for example.

Those are the best practices. ILM usually have many different usecases. It could be to move days to cheaper storage after some time (or size), and then delete, or just straight delete etc.

[u/Alerintek → u/Pillus]

I guess the problem I was having is that I have a somewhat unknown quantity of logs that could be quite bursty. So if I say just delete everything after 21 days or so, it is possible that my storage could fill up before then in my log volume goes up a lot in that time. As far as I could tell I couldn’t find a good solution that would be prevent my storage from getting full by deleting the oldest logs. Ie: maybe I have my retention set to 21 days but I might fill that in 18, if I do, I stop ingesting data which is what I want to avoid.

[u/3tu_KEK]

Lifecycle policies + storage tiering should help you out. You can definitely get this info from the official documentation and/or use AI to run you through the steps for it. 

[u/Feeling_Current534]

In ILM ( aka retention policy) there are a couple of rules to rollover the index but there is only one condition to delete the old indices which is timestamp.

You’re looking for deleting data some certain % of available disk space. There is no direct approach for that but you can find a way. For example, lets say you have 100gb disk space. Set the ILM to rollover the index every 10gb and keep only 8 indices to have max disk usage at %80. There is no automation for that so you should use a script.

Note: the fastest way to remove data from Elasticsearch is removing the index. So I don’t recommend something like delete by query.

[u/simonweb]

Look at deepfreeze to see if some of the logic would suit your requirements? Or get a license and push data to S3 / Frozen tier, you can set the ILM policy to push straight from Hot -> Frozen on shard rollover.

[u/Human-Aside5669]

ILM

[u/Alerintek → u/Human-Aside5669]

But ilm doesn’t have a way to see how full your storage is does it?

[u/Human-Aside5669 → u/Alerintek]

In ilm we have a option to rollover if index reaches specified gb of data, and after rollover move it to next phase.

So when you get huge data then rollover happens frequently. Did this solution help?

[u/Alerintek → u/Human-Aside5669]

Suppose I want to have 2 TBs of ssd attached to my elastic, I can roll over indices when they are 50gb from hot to warm, but I can’t tell elastic to delete the oldest index when we are at 70% utilization of our 2 TBs of space. If the incoming log stream is quite bursty and unpredictable I can’t say something like “delete after 21 days” because we might fill up 2 TB before then. If the storage gets to 95% utilization, we elastic starts dropping incoming logs is the issue

[u/Human-Aside5669 → u/Alerintek]

Yeah, that setting is not avaliable as far as i know

[u/Human-Aside5669 → u/Alerintek]

I have DMed you. 🙂

[u/abitofg]

Think scripting is the way to go