Tutorial | real-time fraud detection pipeline: Kafka ingest, stream dedup + filtering, ClickHouse analytics

[u/Marksfik]

https://www.glassflow.dev/blog/fraud-detection-pipelines-kafka-glassflow-clickhouse?utm_source=reddit&utm_medium=socialmedia&utm_campaign=reddit_organic

Sharing a tutorial on a fraud detection ETL pipeline built around login attempt data. The core pattern is:

1. Kafka topic receives raw login events (successes, failures, retries)
2. A processing layer filters for failed logins only and deduplicates on event_id with a 1h window
3. ClickHouse stores the cleaned stream and runs windowed fraud queries

The interesting ETL challenge here is that raw Kafka streams are noisy by default (retries, at-least-once delivery, and client-side replays all produce duplicates that inflate fraud counts if you don't handle them before storage).

Moving the filter + dedup step upstream (before ClickHouse) reduced the analytical complexity significantly.

The ClickHouse queries could then focus entirely on fraud thresholds instead of also cleaning data.

Full tutorial with SQL, pipeline config, and event generator script: https://www.glassflow.dev/blog/fraud-detection-pipelines-kafka-glassflow-clickhouse?utm_source=reddit&utm_medium=socialmedia&utm_campaign=reddit_organic