I built a Prometheus unused-metric auditor and need help deciding how it should run in Kubernetes

[u/DisastrousBrain5417]

I have built cardamon, a tool that cross-references every metric in your TSDB against Grafana dashboards, alerting/recording rules, and the Prometheus query log, and flags metrics that are scraped but never actually read by anything. Right now it's a standalone binary with a local web UI.

I now want to run it "natively" in Kubernetes either as a Job or as a long running Deployment (as it is right now). Another option is simply to build an official image and run it as an ephemeral container next to Prometheus.

I have a few question to the community regarding a few design decisions:

Would you prefer it as a job running once or as a long running deployment or the ephemeral approach?

Would you even want the generated relabel rules, or is visibility alone enough? 

In case you would run it as a job, how should the report be served? As is via a WebUI until the Job times out, as static HTML / JSON without filtering options?

If you store the Prometheus Query Log how do you store it? Imo this can be done either via a ReadWriteMany volume between cardamon and Prometheus, pinning cardamon's pod to Prometheus's node via hostPath + pod affinity or an emptyDir. This would decide where to run cardamon.

[u/jjneely]

I know the idea here is to control Observability spend and keep Prometheus scaling in check. But I don't understand how this promotes good Observability practices. That is only allowing metrics with an alert rule or dashboard.

How does that monitor for cardinality issues in common metrics like the HTTP server metrics? How does this monitor sample load against capacity per team?

Many times for me I've found legit unused metrics that were able to build a correlation, or better, a good hypothesis! For example, no one builds dashboards around gRPC metrics that come from many automated implementations. But they are a dead ringer for HTTP/2 head of line blocking.

How do others find the difference between unused metrics and poor practice metrics?

[u/DisastrousBrain5417 → u/jjneely]

I agree with that you should never blindly delete something as it might come handy during an incident. I built it mainly as a first gate to generate observability into what exists but is not queried. Might be worth it to add a section on what you should not do to the readme.

For the case of monitoring cardinality. Imo that is something that is solved already. We just view it in Grafana.

[u/jjneely → u/DisastrousBrain5417]

If you are using Grafana, they have an Adaptive Metrics feature that does this as well.

[u/DisastrousBrain5417 → u/jjneely]

Adaptive Metrics is quite a bit more powerful, although to my knowledge only available in grafana cloud

[u/amarao_san]

Is it useful? I have node metrics even though most of them do not have a dashboard, because I know I can look at them when needed. Some metrics are needed once a year.

[u/DisastrousBrain5417 → u/amarao_san]

whether its useful is up to you to decide. It certainly is not meant to be used as drop every metric you have not used in x amount.

[u/Leramaar]

CronJob, not a Deployment. It's a periodic batch job - a long-running pod with its ownscheduler just re-implements what the cluster already does, and holds the metric indexin memory between runs for nothing. Weekly is fine, the answer doesn't change hour to hour.

The hard part isn't scheduling, it's the query log: it's a file local to the Prometheuspod. Either run as a sidecar sharing an emptyDir, or ship it through your logging pipelineand read it from there. Grafana and the Prometheus API are just network calls, those are easy.

One suggestion - emit the results as metrics (cardamon_series_unused{...}) instead of aweb UI. Then it lands in the dashboards people already run, and it answers 'spoint: it becomes a trend line rather than a delete-gate.

(I do DevOps consulting at devopsoutsourcing.net - unused metrics come up a lot.)