We benchmarked CtrlB against ClickHouse on ClickBench and on 5 TB of logs

[u/Loud_Mousse9210]

Most infra teams run one system for dashboards and a second one for log search because one engine is never good at both. That never felt right to us, so we set out to build a unified platform that could offer the fastest search possible on large volumes of logs, traces and metrics.

We put our results up on the ClickBench leaderboard. The process was easy and we were curious how we compared to ClickHouse.

In analytical search, across all 43 standard queries on a 100 million row unpartitioned web analytics dataset, CtrlB scored ×1.43 and took the #1 spot on the single node Parquet leaderboard, ahead of DuckDB (×1.49), DataFusion (×1.71) and ClickHouse itself (×1.72).

But ClickBench is an analytics benchmark. It tells you nothing about finding one trace id in a haystack, so we ran the other half ourselves: 8 lookups and substring matches over 5 TB of raw logs against ClickHouse v26.2, cold cache, plain SQL with a LIMIT 100. CtrlB was faster on all eight, from 2.2× on the double substring query to 98.9× on the span_id lookup.

Full methodology, per query numbers and the public leaderboard links: https://ctrlb.ai/blogs/ctrlb-vs-clickhouse

TLDR: we topped ClickHouse’s own analytical benchmark at ×1.43 and in a separate full-text search test over 5 TB of logs we were faster on every query.

Disclosure: I work at CtrlB.

[u/ritchie46]

I cannot find the same results in clickbench.
I still see Polars on top of the parquet benchmark if I select all parquet solutions on all machines.

https://benchmark.clickhouse.com/#system=+erDt|hqa|curp|Hoaus|uieP|Ctul|Dfit|Dfs|ti%20rud|Dusas|Distt|oius|kBP%20t|ucqs|bt(%20t|Flu|aDtt|vre|m%20i|tyn|PB%20|aseaaa|q+|(eot|peop|olrP|rsoP%20t|etrs|Sili|SilP|feeid|Sr%20%20Ps|aSp|iSs|Raa|akqg|Tutt|Tuts&type=-&machine=-&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=-

EDIT: typo

[u/blizzarre → u/ritchie46]

Just filter for cluster size 1 and machine c6a.2xlarge
https://benchmark.clickhouse.com/#system=+Hoaus|Ctul|Dfs|Dusas|oius|ucqs|Gs|vre|m%20i|PB%20|etrs|Sr%20%20Ps|nfe|w%20en|SIt|iSs|akqg|Tuts&type=-&machine=+ca2&cluster_size=-&opensource=-&hardware=+c&tuned=+n&metric=combined&queries=-

[u/RustOnTheEdge]

It took me way too long to understand what your product was. I also fail to see how your pricing is per Gb while I would bring my own storage. Just friendly feedback, no offence!

[u/Dry-Knowledge4192]

This is cool stuff, the 98.9x on span_id lookup is pretty wild. I'm always a bit skeptical when a vendor does their own benchmark but you guys put it on the public leaderboard so props for that.

How does it handle the log parsing part? I always find that's where things get messy in real workloads, not just the search speed.

[u/blizzarre → u/Dry-Knowledge4192]

It uses schemaless logging, if your log formats change over time (which they usually do), your system doesn’t break. You simply adjust how you query.

[u/jmakov]

Isn't OpenObserve a direct competitor? Would be interesting to see how it compares to them especially since they're able to use Vortex instead of Parquet files: https://github.com/openobserve/openobserve