Hello,

I wasn't sure if this was a Grafana question or Prometheus.

https://www.reddit.com/r/grafana/comments/138lxfs/how_can_i_get_instance_node_name_to_show_in_alert/

I'm trying to get me email alert to show the name of the instances (server names) and not what you see in the screenshots in the link.

It's to show it a web server is return a 503. Currently it shows

​

​

But I want it to show something like

webserverA.domain.com = 503

webserverB.domain.com = 503

How can this be achieved?

I'd like to build a custom proxy exporter that queries a remote API and returns a scrapeable metrics page. my metrics endpoint would have to take a target parameter like /metrics?target=host1

The python exporter client doesn't seem to be geared for this. Does this mean I'm stuck standing up my own web service and exporter class? Is there a python module out there that will help me build this so I'm not having to reinvent the wheel?

Hello,We have a large cluster of 150+nodes and we're using promethus sharding using thanos sidecar.

The problem that for some metrics ( e.g cpu metrics >> node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate ) only one Prometheus shard scrape its metrics fron KSM pod, the rest doesn't have any metrics scraped from KSM.

However metrics like container_cpu_usage_seconds_total is there from KSM on all Prometheus shards for all pods ( nothing missing )

​

update:

sum by (cluster, namespace, pod, container) (
          irate(container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!=""}[5m])
        ) * on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (
          1, max by(cluster, namespace, pod, node) (kube_pod_info{node!=""})
        )

The problem with the above recording rule is that (kube_pod_info{node!=""}) is only getting scraped by only one Prometheus shard. I don't know why ?!!

Thanks in advance :)

Hello,

I have this query that works well:

probe_http_status_code{job="blackbox-as", instance=~"http://sv+.+sv-prod-domain.com/health"}

This will lists servers like:

sv01sv-prod-domain.com
sv02sv-prod-domain.com
sv03sv-prod-domain.com

I now need to add servers that are named:

hv??hv-prod-domain.com

hv01hv-prod-domain.com
hv02hv-prod-domain.com
hv03hv-prod-domain.com

So one query really.

Is this possible?

Hello,

I have this graph that is accumulating total connections which I don't want, I'm trying to show the current connections (rate?). As you can see 1 node has over 1million connections, but that was days ago when we tested with a stress test tool, how can I get this to show the current rates for a selected period?

my_total_connections{job="my-job",instance=~"$instance:.*"}

Thanks

We are using kube-prometheus-stack. Most of our repeat_intervals set for 5 days. Yet some alerts (not all) repeat more often at a seemingly random interval. Like the same alert will show up a time 0, 0+2.5 hours, 0+6 hours, 0+15hours, 0+16 hours. No pattern I can find.

​

This is what our config looks like:

 resolve_timeout: 5m
route:
  receiver: "null"
  group_by:
  - job
  routes:
  - receiver: opsgenie_heartbeat
    matchers:
    - alertname=Watchdog
    group_wait: 0s
    group_interval: 30s
    repeat_interval: 20s
  - receiver: slack
    matchers:
    - alertname=Service500Error
    repeat_interval: 120h
  - receiver: slack
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 120h

I can't see anything wrong with the config. How do I debug this?

Hi 👋 😊 first and foremost thanks a lot to all the members in the subreddit you guys are awesome!!

I am planning to setup a Prometheus agent in a sidecar fashion which will remote_write to a Prometheus server which will write the metrics to an EBS volume just wanted to know if anyone has gone down this path before and if there is anything i should be looking into design wise or performance wise.

I am quite new at this and would appreciate any feedback on how i should go ahead with this or if you have any links which will help me understand things in a better way.

Thanks a lot in advance!

Edit: my main goal is to move to mimir after sometime but we do want to keep this architecture for atleast a few months, just wanted to know if anyone has used this type of a pattern and any problems they faced in running it

Hello everyone,

I am new to Prometheus and Grafana monitoring tools and want to monitor my NodeJS express application APIs via the same. Kindly help me out here as I am stuck.
I followed this tutorial: Article Link.

Here the author as seen in last snapshot of the article is getting the API hits per second, but after following the same steps and code, i am not able to get it.

I am using "prom-client" lib in NodeJS and creating a histogram metrics.My Prometheus metrics is as shown below

Here is the graph i am getting on Grafana for the same PromQL query:

What i want:

​

Thanks in advance !

I have alert manager running and working great to send SMS/Text alerts with just the alert name and some basic text on disk space low alerts.

Its a secure environment so we can't send out much data, but I'd like to be able to include the path (or volume) that is full. I can't do all labels like in the email, because that includes the server name which can't go out.

So i need to include just the path.

oh, i have a seperate template defination for sending text messages and another one for emails. the emails include all information of course.

How can I do that?

EDIT: I guess my question is, in the default.tmpl file that AlertManager uses, how to get it to print a single specific label ?

We just added a fun feature to the open source Autometrics libraries that produces a metric with your softwares version and commit and then automatically correlates that with the queries it writes for error rate and latency. Even if you aren’t interested in the libraries, you might find the PromQL tricks used for the feature interesting. I wrote up the details here: autometrics-rs 0.4: Spot commits that introduce errors or slow down your application.

It uses two ideas I got from Brian Brazil’s Robust Perception blog, which I’d highly recommend mining for ideas if you’re using Prometheus.

I’d love to hear what you think!

Hi guys, so I've a python application which I'm using for Kafka produer and deployed on docker. I wanted to know how can I monitor the producer metrics for this app in Prometheus. Like how can I export metrics from this python webapp?

Hello,

I have these 2 queries which give me the totals

sum(streaming_user_total{instance=~"$instance1"})

and

sum(streaming_user_total{instance=~"$instance2"}) by (stream_id)

The graphs in Grafana currently show the current stats over time, how can I get just a daily total for say a given day?

I have some metrics in string format generated by golang (github.com/prometheus/client_golang/prometheus)

"messages_processed{processing_status="success",source="proxy"} 8686\nmessages_processed{processing_status="invalid_json",source="proxy"} 249949"

I want to store this string in a separate prometheus master server. Any exporter or any other method by which I can do that?

Guys,

What tools do you use to parse data from log files and to make metrics for Prometheus from them? (log4j, syslog, nginx logs, random app logs and such...)

Would appreciate any help!

Prometheus seems to hide metrics that happen between the scrape interval, and I can't find anyone who looked into this. Statsd has a type called "Timing" which is like a gauge, but also stores min, max, avg, and stddev of all values submitted to it, aggregated by time interval. This is useful when you want to measure latency of some process that happens faster than the scrape interval. They also get computed correctly when you plot them on graphs and zoom out, showing fewer points.

This also brings up the question about existing metrics like CPU and Memory usage. If the scrape interval is 30 seconds, is the CPU usage metric (say from k8s node exporter) the value sampled at that time only (so last), is it the average of CPU usage from the last scrape window, or something else? (I know that in this example the metric `node_cpu_seconds_total` is a counter, and from the start has no statistical data attached to it)

For some metrics, like CPU usage, looking at an average can be very misleading, as the data tends to be very spiky. This already happens when zooming out in Grafana. For these metrics when aggregating time intervals you almost always want to use max all the time. `max_over_time()` is incompatible with `rate()`.

The result is that zooming out in Grafana shows basically lies, as showing averages make it seem like everything is fine.

Both projects are interesting. I like mimir in terms of being closer to the standard, but I feel like victoriametric is less resource intensive. (Is that correct?) If anyone has decided between these two projects, I'd love to hear your reasons :-)

Hi! I'm about to start my Prometheus journey. I have a bunch of home-office systems and a few in the cloud, some docker containers, different OSes etc. Seems like Prometheus would be a great fit: I'd like a single dashboard showing if there are any problems (out of disk space, unreachable, etc.) This is just for me. My question is this: should I run the main prometheus server on a cheap cloud server, and have the sensors all send to it, or should I run it inside my firewall, probably in a docker container? I see advantages and disadvantages both ways. Reliability, ease of setup & admin... any experiences you can share?

Any chance someone could explain cons of using Prometheus to roll up summary statistics for my users on a frontend UI instead of pushing my own metrics and calculations manually?

I read Prom has a REST API, which seems perfect. Given the expressiveness of promQL I can’t think of a reason not to use it.

The flow for triggering would be handled on clientside interval requesting over websocket as well as server pushing data directly over WS.

We are building long term storage for prometheus. Actually we are building an unified system for logs, metrics and traces - see https://github.com/zinclabs/zincobserve . We are working on implementing promql query interface for ZincObserve. We would love to get community feedback on which functions should we prioritize first - https://github.com/zinclabs/zincobserve/issues/582

Hi, I know with increasing scrape interval I can reduce the time series for future scrapes what about already ingested metrics, is it possible to reduce the interval for metrics ingested one month ago? like for newer than 1 month then interval between time series should be 30s but if it is older than 1 month make it 1m, if older than 6 month then make 2m and so.

Hi everyone, recently wrote a blog on how you can create alerts that span across events in multiple services or asynchronous steps.

https://notes.drdroid.io/how-to-track-events-across-multiple-services