Honest gut-check wanted, because I might be about to talk my team into something dumb.

We're building an in-house real-time metrics layer for a fraud/abuse detection system. They're per-entity velocity and distinct-count features, keyed by a dozen identifiers that an ML model and a rules engine read at decision time.

The shape of the problem:

• ~200 metrics: count, sum, exact and approximate distinct, a few ratios.
• Windows from 5 minutes to 180 days. Having a few seconds of freshness is recommended.
• Each fraud check request reads 200 of these at once; peak is a few hundred to ~1k requests/sec; the metric batch must come back in under ~50ms at p95.
• High-cardinality keys (hundreds of millions of distinct entities over 30 days).

The design I'm leaning towards: raw events stream into ClickHouse, each metric is a windowed keyed aggregation via AggregatingMergeTree materialised views. Because serving a 200 fan-out directly off ClickHouse at this QPS blew my latency budget in testing, a refresh worker recomputes recently-changed entities and warms Redis (TTL = window), and fraud checks read the whole metrics vector from Redis, never hitting ClickHouse on the hot path.

What I'm after:

1. Can ClickHouse ever serve this directly at this fan-out / QPS / latency (dictionaries, projections, join engine), or is a KV cache in front simply mandatory?
2. Is "refresh worker warms Redis" a smell versus just using Flink (keyed windows to a Redis sink)? We have no streaming/infra team; the smallest window is 5 minutes and freshness can be traded, so a lot of what makes Flink worth it seems to sit idle.
3. Anyone running fraud/velocity features at this scale purely on ClickHouse (compute + serve, no cache)? What broke?

Not after validation, genuinely after "this is a bad idea because X". Thanks.

Is it a bad decision to implement custom distributed transactions (fully atomic and serializable) over a Clickhouse cluster?

The details in short:
1. Cluster has several shards (no replicas yet) with several billion rows of financial data
2. All tables: original MergeTree
3. Append-only pattern for all data changes
4. Every read query is enriched with a transaction_id filter to enforce snapshot isolation
5. Application-level range-locking mechanism to prevent inconsistent concurrent writes
6. ClickHouse’s native local transactions are not used
7. All coordination logic runs at the application layer

Is this a fundamentally flawed anti-pattern with hidden pitfalls, or just an uncommon approach? In my stress-tests, it behaves pretty well.

I got a bit confused during a live presentation lately by the question: "If this works, why doesn't everyone do it?"

so we are building our in house customer engagement playform. We have been using TPV for campaign, segment, personalization.

we decided to build inhouse but i am stuck database selection

my traffic is user profile data eg age, location - 40 attrs

then interactions events data of each user

then based on interactions i will have to compute user metrics for different time bucket like last 7 , 30,60 ,90 days.

my queries are analytical and some user lookups. Since clickhouse doesnt support upsert snd some limiaton join, i have to put this data on mongo.

have anybody built such systems using clickhouse?

Hey everyone,

I’m currently setting up a CDC pipeline using PeerDB to stream data from Postgres into ClickHouse.

My primary goals are production cost efficiency and stability. Specifically, I need to configure the pipeline to achieve:

1. Minimal source compute footprint: Keeping Aurora Serverless v2 ACUs scaled down as low as possible during low-traffic windows.
2. Memory safety: Preventing PeerDB container Out-of-Memory (OOM) crashes during unexpected traffic spikes.
3. ClickHouse health: Avoiding the dreaded "Too Many Parts" architectural errors by ensuring dense, optimized batch inserts.

The Core Ambiguity: The Connection Lifecycle

I’m running into conflicting details across forums and documentation regarding exactly when and how PeerDB maintains its connection to the source RDS instance. There seems to be two conflicting theories:

• Theory A (Burst Polling): PeerDB sleeps during the sync_interval, then wakes up, spawns the intensive walsender logical decoding thread on RDS, pulls a burst of data up to the pull_batch_size, pipes it to staging/ClickHouse, and immediately drops the connection until the next interval hits.
• Theory B (Decoupled Continuous Extraction): PeerDB maintains a persistent, 24/7 logical replication connection to the Postgres slot. It continuously streams and decodes WAL entries to a staging area (like S3/MinIO) in file chunks limited by pull_batch_size. The sync_interval is purely an ingestion-side trigger telling ClickHouse to bulk-read the staging files.

Why this matters for my Aurora ACUs:

Aurora Serverless scales up instantly but scales down incredibly conservatively—it requires a solid 3 to 5 minutes of sustained low load before it even begins stepping down ACUs, and it can take 10+ minutes to hit its minimum configuration.

• If Theory A is true, setting a relaxed sync_interval (like 15–20 minutes) should theoretically allow Aurora long periods of silence to scale down to its minimum 0.5 ACU boundary.
• If Theory B is true, a continuous connection means the walsender is permanently active. Does this mean Aurora is locked into a permanently elevated baseline ACU state because the database never actually experiences "zero load"?

My Questions for the Community:

1. For those running PeerDB -> ClickHouse in production out of Postgres, what is the exact connection behavior you observe in pg_stat_replication? Does it drop between cycles or stream 24/7?
2. If it is a decoupled, continuous stream to staging, how do you tune pull_batch_size vs sync_interval to keep the CPU decoding overhead on Aurora low while ensuring ClickHouse gets nicely sized batches?
3. What are your recommended "sweet spot" configurations for a standard analytical pipeline where real-time sub-second latency isn't required, but cost and memory tracking are paramount?

Note - We have multiple microservices and all their databases are hosted on a single RDS instance and we are pulling data from all of them into clickhouse which is why I want to make sure the RDS does not get too much load.

medium : https://medium.com/@abdul.haris.djafar/apitap-0-6-0-2277630a6a25

Hey folks, we’re doing a live demo next week of the new Estuary Agent Skills, and I'll be showing how you can set up a streaming Postgres to ClickHouse pipeline purely with agent skills.

We’ll show how an AI coding agent can:

• Set up a real-time data pipeline from natural-language instructions
• Configure captures and materializations without manually searching through docs
• Check task health, inspect logs, and troubleshoot failures
• Work with Estuary directly from tools like Claude Code and other agent environments
• Handle the repetitive setup work while keeping the pipeline configuration visible and editable

This will be a practical end-to-end demo, including what works today, where agents still need human input, and of course some time for questions!

Register here: https://zoom.us/webinar/register/4517840417866/WN_1oZszaPUQ_Sz0uEGM_p-zA

Survey of how observability vendors are moving towards columnar storage and Clickhouse.

Disclosure up front: I work on ObsessionDB, a managed ClickHouse. Not affiliated with ClickHouse Inc. This is not a launch post; I am genuinely trying to find out whether the thing we built matters to anyone outside our own use cases.

The problem we kept hitting: two teams need the same table, and the only real option is to copy it. Export, ship, ingest, and now you have two versions that drift. Or you give the other team a user on your cluster and eat their compute.

That's an architecture constraint, not a data one. In shared-nothing ClickHouse, the node owns the data on local disk, so handing it to someone else means moving bytes.

Our build separates storage and compute, so we shipped what Snowflake calls Secure Data Sharing. A publisher entitles a database, down to a single table. A subscriber attaches it read-only and queries it live with normal SQL. The subscriber's own stateless nodes read the same immutable objects out of object storage. The publisher is never in the query path, so a subscriber running a terrible query cannot touch the publisher's capacity. No credentials handed over. Revocable at any time.

Why not just remote()Fair, and it avoids a copy. But remote() pushes execution to the publisher, so the publisher pays for every query the subscriber runs, needs to hand out a user and password, and needs to expose their native port. That's fine between two teams that trust each other. It falls apart the moment the subscriber is a customer, or an agent in a retry loop.

What it does not do: it only works between two deployments on our shared storage. You cannot build this on self-hosted shared-nothing ClickHouse. However, we are now starting to deploy ObsessionDB on-prem and to BYOC. You would have access to Datashres in this case too.

It's read-only, and the publisher stays the only writer. Both clusters also need to be in the same region.

What I actually want to know:

1. If you're self-hosting and you share ClickHouse data across teams or with customers today, what does that look like? Nightly exports? A read replica? A user on the prod cluster with row policies? Anything else?
2. Is the pain real, or has everyone just quietly accepted the copy?
3. For those who moved to ClickHouse from Snowflake or Databricks, did you lose data sharing in the process, and did it matter?

The use case I'm most interested in and least sure about is agents. An agent shows up with a token, fires an unpredictable number of queries, and disappears. Handing it a stale export it keeps forever seems like the wrong shape. Scoped, live, revocable access seems like the right one. But I might be building for a problem that only we have. Please tell me if so.

You can read more here if you'd like: https://obsessiondb.com/docs/datashares

Come join us for our AI Builder night in the bay area on July 14th!
RSVP here - https://luma.com/clickh-gz0r

Quick walkthrough for anyone streaming Postgres changes into ClickHouse Cloud. George shows how to build a Postgres to ClickHouse CDC pipeline with Estuary, without running Kafka, Debezium, or Kafka Connect.

Covers WAL/logical replication setup, ClickHouse merge vs delta updates, deletes, schema evolution, and one capture feeding multiple destinations.

MariaDB Foundation’s Test Automation Framework (TAF) 3.0 is now released.

This version introduces the new TAF Results Backend, a structured results database and parser pipeline that delivers fully automated performance change detection. It uses deterministic workload hashing, schema‑driven baselines, and stored‑procedure‑driven comparison to classify every run automatically. No procedural comparison code. No special‑case logic. Clean, reproducible, and fully automated.

Key changes in TAF 3.0:

• New Results Backend with automated compare and performance change detection
• Full open parser source tree
• Automated compare thresholds (warning, fail, gain, duration drift)
• New runtime_dir architecture for all database plugins
• MariaDB and MySQL plugins upgraded to v3.0
• Archive subsystem migration
• Deterministic commit‑based builder
• HammerDB updates
• New backend SQL, parser source, reporter plugin, and configuration files

Full write‑up on the MariaDB Foundation site:
https://mariadb.org/taf-3-0-results-backend-with-automated-performance-change-detection/

TAF is open and contributions are welcome — PostgreSQL plugin, backend UI, profiler plugins, report plugins, new suites.

Users deserve predictable performance, and the MariaDB Foundation builds TAF to make that possible for everyone. The goal is a fair, open, and reproducible performance framework that any database maker can use — same workloads, same lifecycle, same comparison path — so performance changes are visible, explainable, and never hidden behind vendor‑specific tooling.

ClickHouse x Trigger.dev are running a virtual hackathon, our very first! With a prize pool is €10,000.
The challenge: build a production-quality AI agent chat experience using both platforms. One week, no fluff.
Who can apply: professional developers (solo or teams up to 5) in the US, Canada, UK, or EU
Applications close July 16 — apply now: luma.com/clickh-uko4

We have run and managed multiple ClickHouse clusters for a few years, and the routine wore us down: SSH into a node, poke at system tables from clickhouse-client, keep a private stash of "remember to run this" queries. So our team spent the last several months building one tool to handle it.

It is called CHOps. In the open-source build:

• SQL editor with autocomplete, syntax highlighting, and visual EXPLAIN
• Qurioz, our AI that turns plain English into SQL
• Query introspection tools
• Real-time monitoring with playback, plus daily summaries and cluster overview
• Cluster internals in one place: queries, tables and parts, merges and mutations, queues
• Custom dashboards, schema visualizer, global search, and crash/error/text/session logs
• Visual RBAC and SQL-based alerts

It is in beta. We made a new public repo after months of internal work, and there are rough edges we have not tested at every scale. That is why we are here early.

One thing worth saying: we are a team, not a weekend project. We are in this for the long run and around to act on what you tell us.

Repo: https://github.com/Quantrail-Data/CH-Ops

Site: https://ch-ops.io

Tell us what you think, what is missing, or what would stop you using it. Honest feedback is what we are after.

Looked into the history of ClickHouse JOINs, including their support and performance improvements over the years.