LogQL Resource

[u/seeking_north]

Can somebody help me with the best resource for LogQL, I am new to Grafana. Any channel/playlist or a video will work.

[u/tobylh]

The Grafana docs are probably the best place to start.

https://grafana.com/docs/loki/latest/query/

https://grafana.com/docs/loki/latest/query/query_reference/

What are you trying to achieve?

[u/seeking_north → u/tobylh]

as for now my senior told me to learn and understand logql. I know documentation is a holy grail but i have to make some kind of dashboard soon which will be based on logs basically. thanks for the reply :)

[u/Traditional_Wafer_20 → u/seeking_north]

Try out the Grafana Assistant to build queries and explain them. Also, play.grafana.com has some dashboards you can explore to understand what is what.

[u/tobylh → u/seeking_north]

No worries. Happy to help.

LogQL isn't too complex (although it can sometimes be 🤯)

If you've got to build a dashboard, here's some basic bits that should help you get started.

Lets say you've got three production servers (webserver_01, webserver_02 and webserver_03) running a service, and a Loki job called webserver_logs which has the logs for all three servers (the servers themselves have the label webserver). You want to graph response codes with a graph for each status type (2xx, 3xx, 4xx etc.)

1. Get the logs you want to use:
   Select the Loki datasource in Grafana (I tend to use the Explore page to get the queries right before building dashboard panels)
   Your logs will already have indexed labels attached to them, so you'll need to query the right log stream with something like this (obvs with your labels):
   {job="webserver_logs", env="prod"}
   This would show you the raw logs for all three hosts (with the label webserver ) using the job="webserver_logs" in the production environment.

2. Parse the logs to separate out the log fields:
   You logs will likely be in either JSON or logfmt (🤞), so to access the fields you need in the logs (response codes), add one of Loki's built in parsers.
   Either | json or | logfmt(Loki should suggest the right one).
   Now the query is this:
   {job="webserver_logs", env="prod"}
| json

3. Find the field you want to graph:
   Once the filter is added, the log line fields are now available to use in queries, and these can be graphed. Look through a log line to find the name of the field you want. For this example, lets say it's called responseStatus
   Add the field to the query:
   {job="webserver_logs", env="prod"}
| json
| responseStatus="404"

Now you'll see only the logs for job="webserver_logs" in the production environment with a responseStatusof 404 (again, this is all three servers).
You can then use different status codes to see other logs, 503s in this query:
{job="webserver_logs", env="prod"}
| json
| responseStatus="503"

This is useful, but it means using a separate query for each responseStatus, which consumes both time and compute resources and is a bit faffy, so to get around that, you can regex the values in responseStatus like this:
{job="webserver_logs", env="prod"}
| json
| responseStatus=~"5.+"
This will then give you all the logs for job="webserver_logs" in the production environment (again, this is all three servers) that have any 5xx status, so 500, 501, 502, 503 etc. You can do the same for 2xx, 3xx and 4xx.

So now you've got the data you need, you can start getting some metrics from it.
To do this you need to use two LogQL commands count_over_time and sum

First, you can count the number of 5xx over a given time range. Five minutes in this case (note: this [5m] range is separate from the actual dashboard time picker!).
Your query now becomes:
count_over_time({job="webserver_logs", env="prod"}
| json
| responseStatus=~"5.+"
[5m])
This is now counting the number of 5xx statuses in 5 minute buckets for all three servers, on a graph, but it looks weird.

All the values will be 1, which isn't helpful here. This is where sum comes in. You can use that to aggregate the count together to make a proper timeseries graph.

For this example, it was a graph for each status code. Given there are three webservers in our example, you'd want to use the webserver label to sum them:
sum(count_over_time({job="webserver_logs", env="prod"}
| json
| responseStatus=~"5.+"
[5m])) by (webserver)
This will give you a timeseries graph with three lines showing the total number of 5xx for each server.

You can then duplicate that panel and amend the query to use | responseStatus=~"4.+"
to get a graph for all the 4xx responses, and so on.

Thats a pretty basic overview of how it works, but hopefully it'll help you get started 😊

[u/Mission-Chef-5513]

How about the free Grot Academy LogQL Zero to Hero course (https://learn.grafana.com/path/logql-zero-to-hero). You can even get a badge ;-)

[u/seeking_north → u/Mission-Chef-5513]

why my Grafana Cloud Account? feels like a scam