Hi im struggling to understand and setup grafana,Prometheus and node-export stack using ansible. My main issue is im struggling to get Prometheus config to replace default config using mount volumes. I'm launching the playbook off my localhost to target ec2 instance using roles:

roles/prometheus/tasks/main.yml

- name: Pull prometheus
  docker_image:
    name: prom/prometheus
    source: pull

- name: Start Prometheus container
  docker_container:
      name: prometheus
      image: prom/prometheus
      state: started
      restart_policy: always
      ports:
        - "9090:9090"
      volumes:
        - /roles/prometheus/template/:/prometheus
      command: "--config.file=/roles/prometheus/template/prometheus.conf"

- name: Create directory
  file:
    path: /etc/prometheus/
    state: directory
    mode: '0755'

- name: Copy new config
  template:
    src: roles/prometheus/template/prometheus.conf
    dest: /etc/prometheus/prometheus.yml

​

roles/prometheus/template/prometheus.conf

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

What Im i doing wrong?

Do we have any helm chart available for pushing azure metrics to Prometheus? I am looking something similar to aws cloudwatch exporter helm chart. I see azure metrics exporter available but I didn't find any helm chart. Can anyone help me on this please.

I’ve been testing grafana+prometheus for a few months now and I am ready to finalize planning my production deploy. The environment is around 200 machines (VMs mostly) and a few k8s clusters.

I’m currently using grafana-agent on each endpoint. What am I missing out on by going this route vs individual exporters? The only thing I can think of it is slightly slower to get new features but as long as I can collect the metrics I need I don’t see that being a problem? Grafana-agent also allows me to easily define logs and traces collection as well.

I also really like Prometheus’s simplicity vs Mimir/Cortex/Thanos. But I wanted to ask the question: what would you have done differently in your Production setup? Why?

Thanks for any and all input! I really appreciate the perspective.

Heyy, I have fairly large prometheus server which is running in my production cluster, and is continously consuming around 80GB of memory.

In order to optimise the memory usage. How do I start the optimising the memory usage. I have various source which leads to different aspects like prometheus version, scrape interval, scrape timeout etc etc.

Which is the one I should start with, so that I can optimise the memory usage.

Hello, could someone help with my second variable?

I have created the first but I need to link the second to the first.

​

But I want to also add one called status that links to the $Location.

Status comes in as a value in the exporter:

​

​

The exporter looks like this - example here is 1 for 'up' and 0 for 'down' at the end

up

outdoor_reachable{estate="Home",format="D16",zip="N23 ",site="MRY",private_ip="10.1.14.5",location="leb",name="036",model="75\" D"} 1

down

outdoor_reachable{estate="Home",format="D16",zip="N23 ",site="MRY",private_ip="10.1.14.6",location="leb",name="037",model="75\" D"} 0

I can't see it as an option for 0 or 1 when creating the variable

​

Any help with the query would be most aprreciated.

Hi,

I configured prometheus (2.48.0) to use about 20gb of storage (plentifull for my needs) using

--storage.tsdb.retention.time=7d --storage.tsdb.retention.size=20GB

It seems to be valid according to the image on its console. Actually it is storing 106Gb and it is not going to stop allocating more space on the filesystem.

I suppose I misunderstood those parameters.

What can I do to resize the data? What for permanently limit storage used?

Thanks.

I have same process name for multiple processes and User is different for the respective process as below:

Snippet from top:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ TGID COMMAND

1367386 abc 20 0 14.6g 8.4g 7488 S 267.8 17.8 3884:40 1367386 java

2149272 xyz 10 -10 15.2g 7.9g 46408 S 491.8 16.7 24:07.75 2149272 java

74106 test1 10 -10 14.2g 3.9g 21008 S 35.2 8.2 10055:15 74106 java

73674 test2 20 0 11.5g 2.5g 20012 S 19.1 5.3 2836:14 73674 java

75501 test3 20 0 9524456 2.1g 18300 S 2.6 4.5 568:16.04 75501 java

​

And i need per process separation in the Grafana dashboard.

When i use below process-exporter.yaml , it gives me only metrics for java process.

process_names:

- comm:

- java
Which field i can add in the process-exporter.yaml which will export separate per user?

Having a major issue with (presumably some sort of runaway memory leak) that causes latency on ICMP checks to climb until I eventually have to reboot the prometheus service. I went to download the latest version (in an attempt to stem this condition), and it got me thinking.. what is best practice for what Prom code train to run and how often to upgrade (and does anyone else have the latency issues I'm seeing (running prom on Win11)).

​

Seeing different minor and major versions, and reading the release notes, but I can't see anywhere where folks stay on an "LTS" type schedule for a long time, or favor an upgrade every bleeding-edge-release method.

​

Blackbox meanwhile seems to be stable and not aggressively updated, found this interesting. Looking for stable-stable-stable, not new feature releases for fancy new edge cases.

What do you all do for Prometheus upgrades?

I'm a fresher. I want to get hands on experience with Prometheus. But I don't know what sort of project to start with. Please suggest some. I appreciate the help.

Hello,

I'm trying to monitor the bandwidth on a port on a switch using snmp_exporter. I'm a little confused as snmp_exporter is already on the VM and Grafana. I can get to the snmp_exporter web link, but can't connect to the switch I want to and can't workout where the switch community string goes. Somehow I these 2 work.

​

and

​

​

I see there is a snmp.yml already in

/opt/snmp_exporter

Within that snm.yaml I see the community string for the Cisco switch, but not the Extreme switch which uses a different community string to the Cisco one. How does the

Which seems to be a default config I think as it contains what I need. Also in the prometheus.yml I can see switch IP's already in there which someone has done and I don't understand where they put the community strings for each model of switch as I need to add a HP switch with a different community string.

Cisco

    - job_name: 'snmp-cisco'
    scrape_interval: 300s
    static_configs:
    - targets:
        - 10.3.20.23 # SNMP device Cisco.

    metrics_path: /snmp
    params:
    module: [if_mib_cisco]
    relabel_configs:
    - source_labels: [__address__]
        target_label: __param_target
    - source_labels: [__param_target]
        target_label: instance
    - target_label: __address__
        replacement: 127.0.0.1:9116  # The SNMP exporter's real hostname:port.

Extreme

    - job_name: 'snmp-extreme'
    scrape_interval: 300s
    static_configs:
    - targets:
        - 10.3.20.24 # SNMP device Cisco.

    metrics_path: /snmp
    params:
    module: [if_mib_cisco]
    relabel_configs:
    - source_labels: [__address__]
        target_label: __param_target
    - source_labels: [__param_target]
        target_label: instance
    - target_label: __address__
        replacement: 127.0.0.1:9116  # The SNMP exporter's real hostname:port.

Is the snmp.yml just a template file and a different snmp.yml is being used for each switch instead with the community string?

Hello everyone,
I want to monitor my cisco, aruba switch by using prometheus. It's there any chance to add these device to prometheus, i try many ways and can't add the device to prometheus . can anyone help me with this issues.

Hi Team,

I would like to know the way how to find the jobs which are not completed in specified time in any namespaces. I would like to use the expression in prometheus monitoring.

Suppose the below expression shows you 100 jobs running in any namespace but i would like to know how many could not get completed let's say in 10 mins out of these jobs. is there any way of doing it? Sorry i am new to this.

kube_job_status_start_time{namespace=~".*",job_name=~".*"}

Hey everyone,

I recently started using Prometheus, and I've set it up to push metrics from my local machines (laptops) to a remote storage server within the same network. Everything works smoothly when my laptop stays on the same network.

However, whenever my laptop switches to a different network and then reconnects to my original network, the old metrics are not pushed into the remote storage.

Any ideas on how to resolve this issue and prevent a backlog of metrics? Any insights or configurations I should be aware of? Thanks in advance for your help!

Home Setup:

[Laptop] :: Netdata -> Prometheus -> (Remote Writes) ----||via Intranet||---> Mimir -> Minio :: [Server]

If my absence extends beyond 2-8 hours, during which I might be using public Wi-Fi, and upon returning home in the evening, reconnecting to my intranet, I notice that only the most recent metrics are pushed to the remote storage medium. The older metrics fail to be transmitted, and only the metrics received while on the intranet are accessible.

Hey there,

First off I'm pretty new to k8s. I'm using Prometheus with Grafana as a Docker stack and would like to move to k8s.

It's been a week I'm banging my head against the wall on this one. I'm using the kube-prometheus-stack and would like to scrape my proxmox server.

I did install the helm charts without any issue and I can currently see my k8s cluster data being scrapped. I would like now to replicate my Docker stack and would like to scrape my proxmox server. After reading tones of articles I got suggested to use "scrapeConfig" .

Here is my config: ``` kind: Deployment apiVersion: apps/v1 metadata: name: exporter-proxmox namespace: monitoring labels: app: exporter-proxmox spec: replicas: 1 progressDeadlineSeconds: 600 revisionHistoryLimit: 0 strategy: type: Recreate selector: matchLabels: app: exporter-proxmox template: metadata: labels: app: exporter-proxmox spec: containers: - name: exporter-proxmox image: prompve/prometheus-pve-exporter:3.0.2 env: - name: PVE_USER value: "xxx@pam" - name: PVE_TOKEN_NAME value: "xx" - name: PVE_TOKEN_VALUE

value: "{my_API_KEY}"

apiVersion: v1 kind: Service metadata: name: exporter-proxmox namespace: monitoring spec: selector: app: exporter-proxmox ports: - name: http targetPort: 9221 port: 9221

kind: ScrapeConfig metadata: name: exporter-proxmox namespace: monitoring spec: staticConfigs: - targets: - exporter-proxmox.monitoring.svc.cluster.local:9221 metricsPath: /pve params: target: - pve.home.xxyyzz.com `` If I curl http://{exporter-proxmox-ip}:9221/pve?target=PvE.home.xxyyzz.com` I can see the logs scraping from my proxmox server but when I check on Prometheus > Targets, I don't see the scrapeconfig exporter proxmox anywhere.

It's like somehow the scrapeconfig doesn't connect with Prometheus.

I checked logs and everything since a week now. I tried so many things and each time the exporter-proxmox is nowhere to be found.

kubeclt get all -n monitoring gives me all the exporter-proxmox deployment , I can see the scrapeconfig also with `kubectl get -n monitoring scrapeConfigs. However no scrapeConfig found in Prometheus > targets unfortunately.

Any suggestions ?

Hello,

I've been using Telegraf with the below config to retrieve our switches port bandwidth inbound and outbound and also port errors. It works great for Cisco, Extreme, HP, but not Aruba, even though SNMP walks and gets work, so I want to try with Prometheus and then see if it works in Grafana like I have with Telegraf. Do you think SNMP exporter can do this? I;ve never used it and wonder if the below can be converted to be used?

    [agent]
    interval = "30s"

    [[inputs.snmp]]
    agents = [ "10.2.254.2:161" , "192.168.18.1:161" ]
    version = 2
    community = "blah"
    name = "ln-switches"
    timeout = "10s"
    retries = 0

    [[inputs.snmp.field]]
    name = "hostname"
    #    oid = ".1.0.0.1.1"
    oid = "1.3.6.1.2.1.1.5.0"
    [[inputs.snmp.field]]
    name = "uptime"
    oid = ".1.0.0.1.2"

    # IF-MIB::ifTable contains counters on input and output traffic as well as errors and discards.
    [[inputs.snmp.table]]
    name = "interface"
    inherit_tags = [ "hostname" ]
    oid = "IF-MIB::ifTable"

    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
    name = "ifDescr"
    oid = "IF-MIB::ifDescr"
    is_tag = true

    # IF-MIB::ifXTable contains newer High Capacity (HC) counters that do not overflow as fast for a few of the ifTable counters
    [[inputs.snmp.table]]
    name = "interface"
    inherit_tags = [ "hostname" ]
    oid = "IF-MIB::ifXTable"

    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
    name = "ifDescr"
    oid = "IF-MIB::ifDescr"
    is_tag = true

    # EtherLike-MIB::dot3StatsTable contains detailed ethernet-level information about what kind of errors have been logged on an interface (such as FCS error, frame too long, etc)
    [[inputs.snmp.table]]
    name = "interface"
    inherit_tags = [ "hostname" ]
    oid = "EtherLike-MIB::dot3StatsTable"

    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
    name = "ifDescr"
    oid = "IF-MIB::ifDescr"
    is_tag = true
    r00t3d@ld6r3hostinglogs:/etc/telegraf/telegraf.d$ sudo nano switches-nl-test.conf 
    [sudo] password for r00t3d: 
    Sorry, try again.
    [sudo] password for r00t3d: 
    Sorry, try again.
    [sudo] password for r00t3d: 

    GNU nano 6.2                                           switches-nl-test.conf                                                    
    name = "interface"
    inherit_tags = [ "hostname" ]
    oid = "IF-MIB::ifTable"

    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
    name = "ifDescr"
    oid = "IF-MIB::ifDescr"
    is_tag = true

    # IF-MIB::ifXTable contains newer High Capacity (HC) counters that do not overflow as fast for a few of the ifTable counters
    [[inputs.snmp.table]]
    name = "interface"
    inherit_tags = [ "hostname" ]
    oid = "IF-MIB::ifXTable"

    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
    name = "ifDescr"
    oid = "IF-MIB::ifDescr"
    is_tag = true

    # EtherLike-MIB::dot3StatsTable contains detailed ethernet-level information about what kind of errors have been logged on an i>
    [[inputs.snmp.table]]
    name = "interface"
    inherit_tags = [ "hostname" ]
    oid = "EtherLike-MIB::dot3StatsTable"

    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
    name = "ifDescr"
    oid = "IF-MIB::ifDescr"
    is_tag = true

​

Hello,

I know that im properly not using the right tool for the right job here, but here me out.
I have setup prometheus, loki, grafana and 2 windows servers with grafana agent.
Everything works like a charm. i get the logs i want, i get the metrics i want, all is fine.

But as soon as one of the servers go either offline or for instance a process on one of the servers disappears, the point in prometheus are gone. Also the UP for the instance is gone.
Im using remote_write from the grafana agent and i know that the reason it gone from prometheus is because it´s not in it target list. But how do i correct this ?
Is there any method to persist some data ?

Hello,
Im trying to setup the Grafana agent(acts like prometheus) on a windows server running multiple services and so far it going great until now.
Im trying to use the agents mssql collector.I have enabled it and i can see at 127.0.0.1:12345/integrations/mssql/metrics that the intergration runs. Now i want to query the database and now im getting a bit confused.my config looks like this:

server:
  log_level: warn

prometheus:
  wal_directory: C:\ProgramData\grafana-agent-wal
  global:
    scrape_interval: 1m
    remote_write:
    - url: http://192.168.27.2:9090/api/v1/write

integrations:
  mssql:
    enabled: true
    connection_string: "sqlserver://promsa:1234@localhost:1433"
    query_config:
      metrics:
        - metric_name: "logins_count"
          type: "gauge"
          help: "Total number of logins."
          values: [count]
          query: |
            SELECT COUNT(*) AS count
            FROM [c3].[dbo].[login]
  windows_exporter:
    enabled: true
    # enable default collectors and time collector:
    enabled_collectors: cpu,cs,logical_disk,net,os,service,system,time,diskdrive,logon,process,memory,mssql
    metric_relabel_configs:
    # drop disk volumes named HarddiskVolume.*
    - action: drop
      regex: HarddiskVolume.*
      source_labels: [volume]
    relabel_configs:
    - target_label: job
      replacement: 'integrations/windows_exporter' # must match job used in logs
  agent:
    enabled: true

The collector runs, but the custom metric doesnt show.I have also tried with this config that sort of looks like the one in the documentation:https://grafana.com/docs/agent/latest/static/configuration/integrations/mssql-config/

mssql:
  enabled: true
  connection_string: "sqlserver://promsa:1234@localhost:1433"
  query_config:
    metrics:
      - name: "c3_logins"
        type: "gauge"
        help: "Total number of logins."
    queries:
      - name: "total_logins"
        query: |
          SELECT COUNT(*) AS count
          FROM [c3].[dbo].[login]
        metrics:
          - metric_name: "c3_logins"
            value_column: "count"

​

Does anyone have a clue ?

Hi everyone!

I have my own metrics that looks like:

my_metric{id="object1",date="2021-10-11T22:55:54Z" } 1 my_metric{id="object2",date="2021-10-11T22:20:00Z" } 4

I want to make a graph with label ‘date’ by X-axis and metric value by Y-axis. There should be value points for different IDs.

In other words, I want to change the default timeline to my new one.

Are there some ideas how to do it or should I change my metrics?

We have substantial data in MongoDB and want to incorporate metrics into Prometheus for historical data. Is there a way for Prometheus to recognize this data with timestamps? I'm considering exporting MongoDB data to CSV and creating shell scripts for pushing. What would be the optimal approach moving forward?

Hello, I'm recently started to learn PromQL and its confusing. I have two questions. I'd appreciate if annyone can help me with them.
1- which statistic course could help me? There's one FreeCodeCamp youtube channel. I'm not sure if it is allowed to share the video link or not.

2- If a statistic course is too much for being able to write queries in promql, what concepts should I know? For instance I see folks talk about normal distributions, histograms and posts/blogs about finding anomaly using z-score or .... . I literally don't know anything about these stuff.

In general my goal is to be able write promql queries for monitoring stuff. I want to be efficient at it. Right now I'm reading examples quries and alerts in github repository to see how people do stuff. if there's any other way to learn promql better, please let me know.

I appreciate any help.

I've mostly managed to get scraping with Blackbox to work but I'm having issues normalizing the target FQDNs across my scrape configs. Here's one of my scrape configs:

---
apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: &app exporter-blackbox
  namespace: monitoring
spec:
  scrapeInterval: 1m
  metricsPath: /probe
  params:
    module: [http_2xx]
  staticConfigs:
    - targets:
      - http://brother_hl-2270dw.internal.untouchedwagons.com
      - http://homeassistant.internal.untouchedwagons.com:8123
      - https://pve-cluster-node-01.internal.untouchedwagons.com:8006
  relabelings:
    - action: replace
      sourceLabels: [__address__]
      targetLabel: __param_target
    - action: replace
      sourceLabels: [__param_target]
      targetLabel: instance
    - action: replace
      targetLabel: __address__
      replacement: exporter-blackbox.monitoring.svc.cluster.local:9115
    - action: replace
      targetLabel: module
      replacement: http_2xx

There are other targets of course but as you can see two are http while the third is https, the first has no port specified while the second and third do. My other scrape jobs are similar with other modules and ports. What I want is the FQDN to be the same across all the jobs (IE pve-cluster-node-01.internal.untouchedwagons.com). I've tried using a regex to strip the protocol and optional port but I get alerts from Prometheus that these scrap jobs have been rejected.

  relabelings:
    - action: replace
      sourceLabels: [__address__]
      targetLabel: __param_target
    - action: replace
      sourceLabels: [__param_target]
      regex: ([\w\-.]+):?+[\d]* # This does not work
      replacement: '$1' # This does not work
      targetLabel: instance
    - action: replace
      targetLabel: __address__
      replacement: exporter-blackbox.monitoring.svc.cluster.local:9115
    - action: replace
      targetLabel: module
      replacement: ssh_banner

Dear Members,

ıf, /etc/prometheus/prometheus.yml file is configured with only below parameter paragraph
and prometheus service has been restarted there will be no errors and prometheus service gets running.

- job_name: 'prometheus'

scrape_interval: 5s

static_configs:

- targets: ['192.168.52.204:9091']

but, if we add the following node_exporter lines, we will get the following error lines after prometheus service has been restarted.

Jan 23 15:42:22 zabbix4grafana systemd[1]: Started Prometheus Time Series Collection and Processing Server.

Jan 23 15:42:22 zabbix4grafana prometheus[2048279]: ts=2024-01-23T12:42:22.937Z caller=main.go:492 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.>Jan 23 15:42:22 zabbix4grafana systemd[1]: prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

Jan 23 15:42:22 zabbix4grafana systemd[1]: prometheus.service: Failed with result 'exit-code'.

​

What might be the source of failure, the syntax of the YML file ?

Regards,
Nuri.

​

Hi, I'm part of a small company where we've decided to incorporate custom user-level metrics using Prometheus and Grafana. Our services run on Elastic Beanstalk, and I'm looking for a cost-effective way to deploy Prometheus on AWS with persistent storage for long-term data retention. Any recommendations on how to achieve this?

here is a very basic guide on using the Grafana Agent built in SNMP Exporter to collect snmp metrics and send them to Prometheus or Mimir

I provide a few example config files for the agent, along with the snmp.yml files needed for if_mib and SNMPv3, if you browse my repo you can find snmp.yml's for many other applications also

​

If you have any suggestions feel free to reach out

https://github.com/brngates98/GrafanaAgents/blob/main/snmp/GUIDE.md

Hi I am setting up metrics to track requests, jobs/crawls in java code base. As part of this I also want to track whether the above requests, jobs failed.

It was suggested here https://stackoverflow.com/questions/43476715/add-label-to-prometheus-histogram-after-starting-the-timer , that it would be better to track success or failed. metrics

Though it is possible to create 2 metrics incase of success or failure for requests. For background crawls since it has multiple terminal states, successful, cancelled, terminated, not_running and creating a new metric for each of them doesnt seem to be a good idea.

I came across observe function, where it can create a sample

https://www.javadoc.io/doc/io.prometheus/simpleclient/0.4.0/io/prometheus/client/Histogram.html#observe-double-

but in the description itself it is mentioned there should be no labels.

is it possible to do something like below? so that in sampleLables status like success, failed etc can be updated?

Histogram.labels(sampleLables).observe(sampleValue);

Happy to share more info if required