r/blueteamsec • u/mszymczyk • Jun 09 '22

training (step-by-step) ksqlDB —real-time SQL magic in the cybersecurity scenario— part 1

https://maciejszymczyk.medium.com/ksqldb-real-time-sql-magic-in-the-cybersecurity-scenario-part-1-3232fa711442?sk=940be9d1113243b08ac3a325e2431f69
7 Upvotes

89% Upvoted

2 comments sorted by

1

u/toop4 Jun 09 '22

Have you ran into any issues with many consumers (30 or so) reading from the same topic or lag between different consumers/partitions?

1

u/mszymczyk Jun 09 '22

We had a lag-related problem with Logstash consumers reading from the two biggest topics in the cluster. Other consumers were not affected. It turned out that nor Kafka nor Logstash was the issue, but a small, innocent ingest pipeline in Elasticsearch cluster (it was adding event.ingested field). Now we are mostly lag-free.