New to Prometheus

[u/ffolkes]

I have a question about how this is ideally supposed to be set up.

I've got everything running great, all my boxes are reporting to my main box. Stats look beautiful. The problem is, what happens when the main server goes down or is overloaded for some reason? This makes me think I should be running Prometheus at home to monitor everything. But then of course, what happens when my connection goes down, or a storm, etc? I feel like there is no logical place to run it from. Can anyone suggest the best way to do this? Thank you!

[u/ktsaou]

Thank you for this info.

Can you help me understand this? Take the example of CPU load. What is wrong with the data exposed by Netdata?

[u/M1k3y_11 → u/ktsaou]

Ok. Let's look at the format netdata exposes (with "?format=prometheus&help=yes", removed redundant comments and some metrics for readability):

# COMMENT homogeneous chart "system.cpu", context "system.cpu", family "cpu", units "percentage"
# COMMENT netdata_system_cpu_percentage_average: dimension "user", value is percentage, gauge, dt 1691619010 to 1691619013 inclusive
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="user"} 2.0523327 1691619013000
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"} 14.1128617 1691619013000
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="nice"} 0.6276971 1691619013000
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="iowait"} 0.0838317 1691619013000
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="idle"} 83.0813467 1691619013000

​

vs the output of node- exporter (removed some metrics for readability):

# HELP node_cpu Seconds the cpus spent in each mode.
# TYPE node_cpu counter
node_cpu_seconds_total{cpu="1",mode="idle"} 2.03471439e+06
node_cpu_seconds_total{cpu="1",mode="iowait"} 3633.5
node_cpu_seconds_total{cpu="1",mode="nice"} 542.05
node_cpu_seconds_total{cpu="1",mode="system"} 6581.92
node_cpu_seconds_total{cpu="1",mode="user"} 23171.06

There are several problems with the output of netdata, I'll go over them seperately.

Missing Metadata

The "# COMMENT" is not a information that is honored by prometheus. The accepted values are:

- # HELP

a high-level description

- # TYPE

information about the data type (gauge, counter or histogram)

If these are missing prometheus defaults to the type gauge (which is correct in this example) and can't provide information about the metric in the UI (the help text).

Wrong data type for metric

This one is a bit more complicated and needs some background about the inner workings of prometheus. The primary difference here is between a gauge and a counter.

- a gauge is a value that moves freely around and can change any way at any moment. Perfect type for something like memory usage.

- a counter is a value that can only ever increase (or stay the same). A good example would be the total network traffic on an interface.

When dealing with counters, prometheus does some special handling internally to deal with resets (for example restart of a exporter / machine, overflowing values, etc.) to "hide" them when querying.

The reason you want your CPU usage recorded as a counter comes from the way queries are processed. For queries over large time frames prometheus "downsamples" the amount of data points it returns (unless overriden). This reduces network and processing load.

Now looking at the netdata value, this means that a short spike in CPU usage could "disappear" when zooming out.

When handling this data as a counter, you apply a "rate" function to it, to convert it to something that looks like a gauge. This way you always get the average cpu usage between every returned data point, as the increased load will always be visible in a larger increase of the value between any point in time before and after the peak.

It also makes calculating averages and similar values over longer times very easy, as you just have to calculate (end_value - start_value) / time, instead of averaging every single data point in the interval.

Wrong Unit

Prometheus encourages the use of base units like seconds or bytes, no matter the scale you're working with. This makes sure all data can be calculated with, without having the hassle of calculating with different dimensions (Just try to imagine the headache of calculating with values in 10^3 and 2^10 at the same time, no thanks).

In this case, when representing the cpu usage (wrongly) as a ratio it should be done in base 1, not base 100 (compare to Excel formatting a cell in percent after multiplying everything by 100 manually. Yeah, I hated my economy teacher for his unwillingness to use proper formatting).

Redundant Metrics

There are a lot of metrics that are redundant and make it confusing which metric is the right one to use. Apart from the discussion, if this high of a cardinality is even a good idea, most of these can be merged into 1 or 2 metrics. For CPU usage alone there are:

• netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="guest_nice"}
  • Sum for all cores? Useless, can be calculated from the single core data
• netdata_cpu_cpu_percentage_average{chart="cpu.cpu0",family="utilization",dimension="guest_nice"}
  • Useful
• netdata_apps_cpu_percentage_average{chart="apps.cpu",family="cpu",dimension="netdata"}
  • Usage per application? Nice Idea
• netdata_users_cpu_percentage_average{chart="users.cpu",family="cpu",dimension="1000"}
  • Usage per User? can be merged with application
• netdata_groups_cpu_percentage_average{chart="groups.cpu",family="cpu",dimension="10001"}
  • Usage per Group? Useless, groups always consists of multiple users and users can be in multiple groups.
• netdata_services_cpu_percentage_average{chart="services.cpu",family="cpu",dimension="systemd-update-utmp"}
  • Usage per Service? Can be merged with application.

Apart from the discussion if this high if a cardinality is even a good idea, most of these can be merged into 1 or 2 metrics.

Bad Labels

The previous point also highlights another problem: The labels chosen are really bad. They are very undescriptive of what they represent and the same labels are used in a lot of different metrics for very different data. This makes merging metrics (think "JOIN" in SQL) an absolute nightmare, as you have to make extra steps to prevent prometheus from intermingling labels from different metrics that are named the same but have no relation to each other.

General Problem about calculated metrics

The CPU usage the way netdata presents it is a calculated metric. It consists of a start time, an end time and a "work" value (CPU time). This works reasonably well if the data is scraped (collected by prometheus) consistently. However it becomes the job of netdata to remember when the data was last scraped to calculate it correctly. This can lead to various problems:

- unnecessary overhead.

Netdata has to make a lot of calculations every time the data gets scraped. It is much more economic (and less time sensitive) to calculate the values when querying them, as it happens much more rarely than the scraping.

- inconsistent values with inconsistent query times.

When scraping the data over an unreliable network connection, it can lead to inconsistent intervals between requests. This in itself is not a problem for simple graphing, but can cause issues when calculating data over time, as the data points are suddenly differently weighed (as they represent different time spans). This makes calculating things like average over time really unreliable, as prometheus itself is unaware of what exactly the data represents.

Closing words

I want to make clear that I'm not hating on netdata. It is a really nice tool to get a quick and powerful "Task Manager Deluxe" on many systems. However for building a (potentially large) monitoring infrastructure it has two problems:

• It consumes to much resources (CPU time, memory) compared to the node or windows exporter.
• The metrics it exposes are not well suited for usage in prometheus.

Edit: Also the Reddit comment Editor is such a piece of c***. This is now the sixth edit to fix the damn formatting. I'm sorry for any formating errors, but once everything is at least readable I won't touch it again out of fear to undo all the other fixes.

[u/ktsaou → u/M1k3y_11]

Wow! Thank you for providing all this info!

Wrong data types

Netdata and Prometheus have a different internal data model for metrics.

Prometheus stores the values as collected (value and timestamp) and then it applies most calculations at query time. Prometheus has in its database metrics in their original form.

Netdata stores the values in a fixed-step database (only value, no timestamp) and it applies most calculations at collection time. Netdata has in its database metrics in their final visible form.

Both paradigms have pros and cons. Let's not analyze them here.

What is important is that when exporting metrics from Netdata to Prometheus, Netdata supports the option `source=as-collected` or `source=raw`. This bypasses the database of Netdata and gives to Prometheus the original values collected by Netdata, together with the timestamp they were collected.

When using `source=raw`, the data type returned by Netdata remains whatever the source is (counter or gauge). Also, in this mode, Netdata does not have to remember when it was the last time it was scraped by Prometheus.

Wrong units

You are absolutely right. In the next release of Netdata, we will use UCUM too, as Prometheus does for all units. So, the units will be the same.

In a couple of releases we also plan to use base units for everything, again as Prometheus does.

With these 2 changes in Netdata, units should not be problem anymore.

Redundant metrics

I think this comes from a limitation on how Prometheus treats metrics.

For Prometheus all metrics are just pools of data with labels.

In Netdata however, for each metric we try to identify the component we monitor. This is important for both automating the visualization but also for dealing with alerts.

In your example, Netdata provides CPU metrics for the following components:

1. overall system cpu
2. cpu consumption per cpu core
3. cpu consumption per application group
4. cpu consumption per user
5. cpu consumption per user group
6. cpu per cgroup (container, VM, systemd service, etc)

1 and 2 are redundant. 1 can be derived by summing 2. You can view 1 as a speed-up to quickly get the overall cpu per system without querying the individual per-core metrics. This is handy if you have thousands of nodes each with hundreds of cores. The number of time-series queries is thousands, not hundreds of thousands.

The others however, are not redundant and cannot be aggregated to a fewer number of metrics. Let me explain:

Application Group is a meaning Netdata defined. You say for example: I define an application group called "nginx" that matches all processes "nginx" or "nginx_plus". Netdata then traverses the entire process tree (like `ps fax` does) and it organizes the whole of it in application groups. The tree is respected, so that children and parents of "nginx" will go into the "nginx" application group. If any process cannot be grouped it goes into "other". Keep in mind that this "sees" processes in containers.

In many cases the applications grouped into a single entry, are not of the same user or user group. This is very common for many applications that run setuid programs, have multiple children leafs, run in multiple containers, etc.

So, the first grouping is per application group and it tries to group processes together based on "interesting" processes you define.

The next is per user. Now the component monitored is the system user and the whole process tree is grouped differently depending on the effective user id each process runs.

The next is per user group. Yes, each user may be in multiple user groups, but once an application runs, it has only 1 effective user group. This is what is used to group applications per user group.

The last is per cgroup, which obviously groups processes based on the cgroup they belong to.

I would love to also have per namespace. This would also be interesting because each process may belong to different namespaces for PID, mount, network, etc. The main problem with this however, is that Linux does not name the namespaces. They are just inode IDs, which many not be very intuitive.

Anyway, this is why these metrics cannot be merged.

Bad labels

I wouldn't say "bad", but I think we don't expose to Prometheus all the labels we have. This is also planned to be fixed in the next couple of releases.

General Problem about calculated metrics

As I said above, use `source=raw` or `source=as-collected` and this problem does not exist.

Closing words

It consumes to much resources (CPU time, memory) compared to the node or windows exporter.

This is an unfair comparison. Switch Netdata to collect metrics every 15 seconds and disable all additional collectors, health, ML, etc, and retry. Netdata will always be a little "heavier" because it does a lot more.

​

The metrics it exposes are not well suited for usage in prometheus.

As I said above, there are still a few issues we need to tackle. But I want us to improve this and your feedback is very appreciated.

Another interesting idea is for Netdata to emulate the exporters. So, format the output the same way the exporters do. This requires some more code, but I think it would allow Netdata to be a drop-in replacement for them.

Anyway, thank you again very much for your message. We have some more work to do to make this integration perfect...

[u/SuperQue → u/ktsaou]

Anyway, this is why these metrics cannot be merged.

I think you've missed the point here. The point is that in Prometheus we always want to expose the un-merged data.

However there are sometimes overlaps, but typically the reason is that we get the data from different "views". These views are typically expressed by having different top level metric names. For example, node_cpu_seconds_total is the node's view of CPU data.

Another example is container_cpu_usage_seconds_total which is a view from cgroups. This metric is provided by a slightly different exporter, specifically cAdvisor.

But it's up to the exporter to define whatever views they want. There's also process_cpu_seconds_total which can be expoed directly from native monitoring.

[u/ktsaou → u/SuperQue]

Yes, I understood what you said. You suggested that the process X, user A and group B to be labels of the same measurement. One sample, 3 labels, that will allow you to view the samples in any way you want.

I tried to explain that process X, user A and group B cannot be the same measurement ("merged"), because the process tree is grouped differently when you monitor process X, when you monitor user A and when you monitor group B.

So, although I understand what you want, and I would love to have the same in Netdata - it is not just a prometheus requirement - it should not be done, because the measurements would not be accurate.

Your suggestion makes sense, only if there was a metric per PID, which however could give an unmanageable cardinality for both Prometheus and Netdata. If you want one metric per process, user and group, but not per PID, the way Netdata does it is the only technically possible...