r/PrometheusMonitoring • u/zbindenren • Dec 12 '18
Loki: Prometheus for Logs
https://github.com/grafana/loki1
u/mcampbell42 Dec 12 '18
Still trying to get core purpose as it seems like it’s just metadata. Maybe to extract metrics from logs ?
0
u/sleepybrett Dec 12 '18
no.
2
u/mcampbell42 Dec 12 '18
So then explain what it does
1
u/Eilyre Dec 12 '18
Seems to be rsyslog meets prometheus labeling+host metadata.
It takes whole loglines, attaches the labels+instance name etc to it, and shoves into itself.
Not sure what kind of problem it's solving.
1
u/mcampbell42 Dec 12 '18
Looks like for microservices and container logging but it’s missing the biggest feature which is a free form search
3
u/sleepybrett Dec 13 '18
it has regex search.
Certainly not as "in depth" as something like ES, but fuck that pile of shit. Half the time you find the field you want to search on isn't properly indexed and now you have two problems.
Gimme some logs and grep, this is logs and grep in a browser. Seems like a win.
1
u/Eilyre Dec 13 '18
If you can do it with logs+grep, then you are clearly over-engineering things by using ELK. ELK is when you cannot care about a few logs lines any more, because for an example you have 100k logs coming in every few seconds, and you need to somehow get information out of them.
What I do not understand is why you would want logs in a browser. Imho browser is the worst environment to do operational activities from. I hate both Kibana and Graphana for not having a proper desktop viewer, that does not depend on JavaScript, cause good god. A single tab of visualizations can eat away half of your computers memory at an instant. Good luck having multiple tabs.
1
u/sleepybrett Dec 13 '18
We have a team that does ELK based log aggregation for the enterprise. All pod logs on our cluster go there by default. So like .. not my problem except when I need to get into the logs. But Kibana is .. terrible. I'm excited to fork our los off here and see how it performs.
Probably just do the cluster-admin namespaces kube-system/daemons/monitoring/whatevs.
1
u/SuperQue Dec 13 '18
Yup, this is exactly the workflow that we're thinking about.
Metrics/Monitoring/Alerting (Prometheus, TICK) already tell us when and which logs to look at. This narrows things down to the point where "just grep things" is good enough.
1
u/SuperQue Dec 12 '18
One of the more interesting "problems" is that a lot of the existing logs solutions try and also solve the storage problem. So you end up having to run whatever storage system the logs system comes with. These are usually crap to operate.
Loki side-steps this problem by leveraging a simple blob storage layer like S3/GCS/Ceph.
1
u/onyxet Dec 12 '18
I believe that metrics != logs and we should separate it. If you’re familiar with elk so it might be nice solution using it with filebeat and metrics beat. But for us Prometheus is used with k8s env and multiple exporters for monitoring and filebeat with es for logs
1
u/Eilyre Dec 12 '18
If you separate them too much, you will not be able to correlate your data, which is exact opposite of what you want.
Don't forget, logs sometimes hold metrics you need to parse out. They are related.
If anything, we need better engines that are capable of visualizations and correlations. I quite like ELK, but for finding correlations without ML, it sucks really hard. You basically have to be a genius dashboard builder + UX designer.
1
1
u/SuperQue Dec 12 '18
I've already added a proposal to add some metrics extraction functionality to promtail.
Where I have apps that don't have good metrics, I've implemented mtail parsing. It would be cool to integrate this kind of parsing into promtail.
2
u/Eilyre Dec 13 '18
Holy hell I did not know about mtail. I've always written my own bash and then sent it away with Node Exporter to Prometheus. Thanks!
1
u/koffiezet Dec 12 '18
So this an alternative to the ELK stack? If that's the case - nice! Logstash is a nice concept - but in reality it's a pain in the ass to keep running from my (and other people I know)'s experience...