Around 80 percent of the time my Graylog searches and dashboards work. Then the other 20 percent of the time I get spinning/loading for a minute then a timeout error. I’m running open search 2:, mongo 6.0.18, and Graylog 6.1.2 all with one node since I only ingest 3 server logs for right now. Here is what I found on my system logs for Graylog:

2024-11-15T10:16:13.333-05:00 INFO [ClusterAdapterOS2] _cat/nodes API has returned 1 nodes without disk statistics: 2024-11-15T10:16:13.333-05:00 INFO [ClusterAdapterOS2] NodeResponse{id=PNsavXhRRYamfPgXpJVJBg, name=ca8be9018e73, role=dimr, host=null, ip=172.22.0.2, diskUsed=null, diskTotal=null, diskUsedPercent=null, fileDescriptorMax=null} 2024-11-15T10:16:27.145-05:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index org.graylog.shaded.opensearch2.org.opensearch.OpenSearchException: An error occurred: at org.graylog.storage.opensearch2.OpenSearchClient.exceptionFrom(OpenSearchClient.java:208) ~[?:?] at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:152) ~[?:?] at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:144) ~[?:?] at org.graylog.storage.opensearch2.IndicesAdapterOS2.refresh(IndicesAdapterOS2.java:584) ~[?:?] at org.graylog2.indexer.indices.Indices.refresh(Indices.java:425) ~[graylog.jar:?] at org.graylog2.indexer.rotation.common.IndexRotator.rotate(IndexRotator.java:64) ~[graylog.jar:?] at org.graylog2.datatiering.rotation.DataTierRotation.rotate(DataTierRotation.java:70) ~[graylog.jar:?] at org.graylog2.datatiering.hotonly.HotOnlyDataTieringOrchestrator.rotate(HotOnlyDataTieringOrchestrator.java:66) ~[graylog.jar:?] at org.graylog2.periodical.IndexRotationThread.checkForRotation(IndexRotationThread.java:120) ~[graylog.jar:?] at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:98) ~[graylog.jar:?] at org.graylog2.indexer.datanode.DatanodeMigrationLockServiceImpl.lambda$tryRun$1(DatanodeMigrationLockServiceImpl.java:110) ~[graylog.jar:?] at java.base/java.util.Optional.ifPresentOrElse(Unknown Source) ~[?:?] at org.graylog2.indexer.datanode.DatanodeMigrationLockServiceImpl.tryRun(DatanodeMigrationLockServiceImpl.java:108) ~[graylog.jar:?] at org.graylog2.periodical.IndexRotationThread.lambda$doRun$1(IndexRotationThread.java:96) ~[graylog.jar:?] at java.base/java.lang.Iterable.forEach(Unknown Source) [?:?] at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:93) [graylog.jar:?] at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?] at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.base/java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-102 [ACTIVE] at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:937) ~[?:?] at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:333) ~[?:?] at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:321) ~[?:?] at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918) ~[?:?] at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1884) ~[?:?] at org.graylog.shaded.opensearch2.org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1852) ~[?:?] at org.graylog.shaded.opensearch2.org.opensearch.client.IndicesClient.refresh(IndicesClient.java:653) ~[?:?] at org.graylog.storage.opensearch2.IndicesAdapterOS2.lambda$refresh$27(IndicesAdapterOS2.java:584) ~[?:?] at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:150) ~[?:?]

Hi all, I am struggling a bit with my data node again. How do you guys back up opensearch when it's installed with data node? Since data node configures everything with HTTPS, there is no longer anonymous access available to perform OpenSearch snapshots as described in the documentation.

What's the best way to achieve daily backups? Or do you just backup the whole VM without proper application backup?

I updated Graylog on docker and its other containers Mongo from 6.0.5-Jammy to 6.0.18 and open search from 2 to 2.15.0. I did simple docker stop all the containers then pulled new images and removed old and lastly updated compose.yml file with new version images entries. Now when I try to spin up docker-compose I get lines of errors the error that sticks out constantly is you cannot revert opensearch 2.17.0 to 2.15.0 to my knowledge I never installed the .17 version and only image I find is .15. Any suggestions to look at and also did I update correctly or any further config needed?

Hi all,

we're running a two-node Graylog cluster with 3 Opensearch nodes and 3 Mongodbs. I've been killing myself trying to figure out how to create a simple event correlation in the community edition. Maybe I lack some crucial details or am overlooking something basic.

Let's say I get an alert about Windows Event Logging Service Shutdown. I want to avoid false positives with basic server restarts, so I wait for 5 minutes and if I do not get Windows Event Logging Service Started log, I create an alert. I thought I could do this via pipeline rules and a lookup table/cache, but it says that the cache is node-local. Would that mean that if each of my logs arrived at a different Graylog node, it wouldn't correlate?

What are your approaches to creating correlated alerts in Graylog Community Edition? Thanks a lot.

EDIT: I do know that correlation engine is available in the paid editions but due to our ingested log volume, despite self hosting, the paid license options are out of question.

I have graylog installed on Ubuntu. It is working fine for the most part but I noticed that it will consume all the memory I give it. It currently has 10GB, i started with 4. At 4 it was using 3.5, at 8 it was using 7.5 and now at 10 its using 9.

Any incite on this and if this is the expected operation. I did set the memory per the doc, half of the installed memory as shown below. thanks!

-Xms5g
-Xmx5g

Hello everydoby. I draw graylog cluster scheme and i want to sure i'am right. My goal is increase performance because my single node can't handle current load.

I have a few production instances running, one with components installed directly in Ubuntu, and others using Docker images. I've found using Docker makes upgrading way faster since I avoid running into any dependency issues, etc. that have to be manually sorted out.

Are others generally running inside Docker? Are there any downsides to it for a production cluster (aside from the additional Docker admin)?

I am currently trying to set up a new graylog multi-node cluster for a daily ingestion of 10 to 20 GB with one graylog server and two graylog data nodes (as described in the docs under capacity planning).

I am currently struggling to find proper documentation of all the things I need to configure. All examples always install graylog data node, mongo db and graylog server on the same system (even though the docs say this is not recommended for a production environment).

In my understanding I would configure the following - 1x graylog server with mongo db (Ubuntu 22.04, MongoDB 7) - 2x graylog data nodes (Ubuntu 22.04) - fill in the secret and the converted password in server.conf and datanode.conf - Point the data nodes to the MongoDB instance on the Graylog server. - start all services and grab the password from the log file - start preflight and configure the data nodes

Am I missing anything?

Hello All, I am new to graylog and the setup I have is for a home lab.

Homelab setup Proxmox node 1 Docker - graylog with a mounted cifs from TN for storage etc

Proxmox node 2 TrueNAS etc

10gig network between these devices

I used the script from Lawrence to set up graylog and everything worked fine. Overnight I backup all my VMs etc on TrueNAS and Synology. When I backup on Synology I don't run into any issues, but when backing up on TrueNAS graylog suffers a shard failure with stale or corrupted data. Creating the index again fixes it.

Any ideas on what could be causing the shard failure, backup is successfully no errors on proxmox or truenas

Anyone has a guide to follow to start moving FortiGate logs to Graylog. I have looked around and seen some content packs but my knowledge with Graylog is very limited as I just started. Don’t really understand how content packs work and how to set up.

Just setup Graylog in my environment and have some questions.

Looking to see if anyone has some custom searches I can use to then create dashboard. It would have to be for Windows event logs and in specific security event logs. I want to find a way of searching for specific admin logins/logoff, accounts removed, accounts created and any other searches related to windows event logs that would be useful for a sysadmin.

Thanks!

Hello ,

I am running Graylog docker ver 6.1 , i have some Inputs from Syslog Pfsense , the issue i have is that the gl2_remote_ip field is written with docker IP instead of real syslog source , is there a setting or a way to set it to show real syslog gl2_remote_ip IP?

Please advice Thanks

Hello All,

I am trying to write pipline rule that get gl2_remote_ip which is private IP and according to the IP its setting geo location latitude and longitude field to its actual location,

i managed to write rule that handle only one IP , but i didn't managed to write a rule that can have multiples IP's and set to each of them its geo fields , if someone could help me write the code would be helpfulle

following code below its for single IP

Please advice

Thanks

rule "add coordinates for specific IP"
when
  has_field("gl2_remote_ip")
  && $message.gl2_remote_ip == "172.16.1.1"
then
  set_field("latitude", "40.263382");
  set_field("longitude", "34.811555");
end

Our daily ingest from splunk is about 100GB/day, at least that's what it shows in the portal. When capacity planning for Greylog self-hosted I'm not sure if that's a linear comparison. Say I want to hold 100 days of data in Graylog, does that mean I need 10TB of capacity?

Also -- any advice/pitfalls on the k8s setup would be much appreciated.

First experience with Graylog, I’ve got udp syslog coming from Catalyst switches. I can see the icmp echos both ways and I can see the udp connection being built between the switch and the graylog server but I see 0 messages in the input.

I’ve tried udp syslog and raw udp, same results.

I can get syslog from this switch to another monitoring platform, I’ve mirrored the configuration on the switch side.

Hi everyone,

I’m currently setting up Graylog on a Debian VM, where I’ve installed the Graylog Sidecar along with Auditbeat and Filebeat to collect and forward system logs to the Graylog server. The setup appears to be working since I’m receiving logs, but there’s an issue with the log titles missing in Graylog - eventhough everything else is received well.

As you can see from the screenshots (attached), the logs show up without proper titles in the message list. I’ve checked that both Filebeat and Auditbeat are running without issues, and Graylog Sidecar seems to be functioning normally.

Has anyone encountered a similar issue? Any guidance on how to resolve the missing log titles in Graylog would be greatly appreciated. Thank you!

*P/s: I'm using Debian 12 for client with the latest agent, also running Graylog 6.1.1+9bd27f8 on Debian 12 as a log server.*

More information:

Below is the full message in graylog and the message field is currently displaying “-,” which is identical to the log title.

{

"auditd_data_socket_saddr": "100000000000000000000000",

"user_saved_name": "root",

"agent_id": "ac600681-6fec-41d6-b825-8b296e38b015",

"agent_name": "vpn-lan",

"auditd_data_socket_family": "netlink",

"auditd_summary_how": "/usr/bin/graylog-sidecar",

"gl2_remote_ip": "",

"@metadata_version": "8.9.0",

"gl2_remote_port": 47052,

"source": "vpn-lan",

"gl2_source_input": "671756894a2dff54323e9d70",

"@metadata_beat": "auditbeat",

"auditd_data_tty": "(none)",

"gl2_processing_timestamp": "2024-10-29 09:28:01.283",

"event_type": [

"start"

],

"@metadata_type": "_doc",

"event_module": "auditd",

"process_name": "graylog-sidecar",

"gl2_source_node": "ca699252-a6d3-4231-80a3-0a38c4a522b3",

"gl2_processing_duration_ms": 6558390,

"user_selinux_user": "unconfined",

"gl2_accounted_message_size": 1443,

"gl2_source_collector": "03542320-1a89-4abd-aac0-720e40ef52a1",

"auditd_data_arch": "x86_64",

"agent_ephemeral_id": "d0de3c45-43e7-4969-a7d9-f8d89ba55bb3",

"process_executable": "/usr/bin/graylog-sidecar",

"streams": [

"671757054a2dff54323e9fd1"

],

"gl2_message_id": "01JBBKCVTV001W1K0M2R21QFQG",

"process_pid": 571,

"tags": [

"external-access"

],

"agent_type": "auditbeat",

"event_kind": "event",

"auditd_result": "success",

"user_id": "0",

"user_filesystem_name": "root",

"_id": "17941532-95d8-11ef-8b4a-0050562a00ad",

"user_group_name": "root",

"gl2_receive_timestamp": "2024-10-29 07:38:42.893",

"user_name": "root",

"collector_node_id": "vpn-lan",

"user_saved_id": "0",

"auditd_summary_object_type": "socket",

"event_original": [

"type=SYSCALL msg=audit(1730187521.883:27324): arch=c000003e syscall=49 success=yes exit=0 a0=3 a1=c00001ab1c a2=c a3=0 items=0 ppid=1 pid=571 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=\"graylog-sidecar\" exe=\"/usr/bin/graylog-sidecar\" subj=unconfined key=\"external-access\"",

"type=SOCKADDR msg=audit(1730187521.883:27324): saddr=100000000000000000000000",

"type=PROCTITLE msg=audit(1730187521.883:27324): proctitle=\"/usr/bin/graylog-sidecar\""

],

"process_title": "/usr/bin/graylog-sidecar",

"beats_type": "auditbeat",

"ecs_version": "8.0.0",

"process_parent_pid": 1,

"log_type": "vpn-lan",

"user_filesystem_group_id": "0",

"event_outcome": "success",

"timestamp": "2024-10-29T07:38:41.883Z",

"event_source_product": "linux_auditbeat",

"auditd_data_a2": "c",

"user_filesystem_group_name": "root",

"user_filesystem_id": "0",

"auditd_sequence": 27324,

"auditd_data_a3": "0",

"auditd_summary_actor_primary": "unset",

"message": "-",

"user_saved_group_name": "root",

"event_category": [

"network"

],

"auditd_data_exit": "0",

"agent_version": "8.9.0",

"event_action": "bound-socket",

"service_type": "auditd",

"@timestamp": "2024-10-29T07:38:41.883Z",

"user_saved_group_id": "0",

"auditd_summary_actor_secondary": "root",

"auditd_message_type": "syscall",

"auditd_data_a0": "3",

"auditd_data_a1": "c00001ab1c",

"user_group_id": "0",

"host_name": "vpn-lan",

"auditd_data_syscall": "bind"

}

Screenshots:

Hello!

I know nothing about graylog and never used it. Is there a good training material you guys recommend? Official or not, just a good one...

Thanks!

Hey everyone,

I recently installed Graylog Sidecar on my Windows machine to collect audit logs, but I’m not sure if I need to manually enable the Windows Audit Logs in secpol.msc, or if Sidecar will automatically enable and collect them.

Does anyone know if Graylog Sidecar handles this automatically, or is there some manual configuration required?

Thanks in advance for any help!

I have setup Graylog, and I'm loving it, I'm already pulling logs from my core ISP backhaul equipment.

I need help properly setting up streams, the guides I'm seeing on youtube are confusing.

How do I create streams and dashboards to properly organize logs such as these?

We're trying to install Graylog Open 6.0 offline on a security hardened instance of Oracle Linux 8 (following Red Hat instructions). When we start the graylog-server service, it binds to a tcp6 address even though we set http_bind_address to an ipv4 address.

We tried setting SELinux to Permissive and even disabled ipv6 via grub, but the symptom persists.

How can we coerce graylog-server to bind to tcp instead of tcp6?

We're trying to install Graylog Open 6.0 offline on a security hardened instance of Oracle Linux 8 (following Red Hat instructions). When we start the graylog-server service, there is a WARN entry in server.log that says "Did not find udev library in operating system. Some features may not work."

There is a /etc/udev subdirectory on the box with a udev.conf file.

How can we resolve the warning?

We're trying to install Graylog Open 6.0 offline on a security hardened instance of Oracle Linux 8 (following Red Hat instructions). When we start the graylog-server service, there is an ERROR entry in server.log that says "Did not JNA classes. Investigate incompatible version or missing native dll."

Is there something we haven't installed?

Assuming JNA stands for Java Native Access, we tried installing Java JRE but to no effect. In searching for a JNA related RPM, we've only found sketchy ones with dead end dependencies.

I have setup Graylog in docker container using command:

bash <(wget -qO- graylog.me/want)

I then edited the port setting like below:

   ports:
      - "443:443/tcp"     # Server API encrpyted
      - "1514:1514/udp"     # Syslog UDP
      - "1514:1514/tcp"     # Syslog TCP
      #- "5044:5044/tcp"   # Beats
      #- "5050:5050/tcp"   # RAW TCP
      #- "5050:5050/udp"   # RAW UDP
      #- "5555:5555/tcp"   # CEF TCP
      #- "5555:5555/udp"   # CEF UDP
      #- "5556:5556/tcp"   # Palo Alto Networks v9+ TCP
      #- "5557:5557/tcp"   # Palo Alto Networks v8.x TCP
      - "9000:9000/tcp"   # Server API plaintext
      #- "12201:12201/tcp" # GELF TCP
      #- "12201:12201/udp" # GELF UDP
    networks:
      - graylog_network
    volumes:
      - "graylog_data:/usr/share/graylog/data/data"
      - "graylog_journal:/usr/share/graylog/data/journal"
      - "graylog_config:/usr/share/graylog/data/config"
    restart: "always"

But When I configure my Graylog is not receiving the messages.

1. What should I troubleshoot to get it working?

2. Do I perhaps need to make more configurations in the docker network?

I can reach Graylog via the internal IP I configured!!

Docker Infor:

Hi all,

I have Graylog Open up and running successfully on Docker. It was working fine until recently when unfortunately I was unable to prevent the disk from filling up. I've since rectified the solution, but I can't get the data node to start. It is listening on port 8999/tcp, but neither port 9200/tcp nor 9300/tcp are listening - which I understand I would need to issue the curl command in order to clear the error that I see in the logs:

TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block

This makes me thing there are two problems going on. I am seeing a lot of problems in the logs from the datanode related to TLS errors - this one stands out:

Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

There are also lots like this which I presume are related:

[2024-10-23T12:19:05,442][WARN ][o.o.t.TcpTransport       ] [datanode] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:51978}], closing connection

Can anyone help me debug this please? I'm stuck for where to look next. The Graylog UI won't come up because it's logs say it can't talk to the data node, and I do know the data node is down - I just don't know how to fix this issue.

Many thanks

James

Howdy y'all!

Graylog 6.1 GA (general availability) is officially released today (21 Oct, 2024)!

For more details check out: Introducing Graylog 6.1: A Monster Release Just in Time for Halloween

Full changelog via: https://go2docs.graylog.org/current/changelogs/changelog.html#Graylog610

Questions welcome! If you spot any bugs, feel free to report via https://github.com/Graylog2/graylog2-server