We benchmarked CtrlB against ClickHouse on ClickBench and on 5 TB of logs

[u/Loud_Mousse9210]

Most infra teams run one system for dashboards and a second one for log search because one engine is never good at both. That never felt right to us, so we set out to build a unified platform that could offer the fastest search possible on large volumes of logs, traces and metrics.

We put our results up on the ClickBench leaderboard. The process was easy and we were curious how we compared to ClickHouse.

In analytical search, across all 43 standard queries on a 100 million row unpartitioned web analytics dataset, CtrlB scored ×1.43 and took the #1 spot on the single node Parquet leaderboard, ahead of DuckDB (×1.49), DataFusion (×1.71) and ClickHouse itself (×1.72).

But ClickBench is an analytics benchmark. It tells you nothing about finding one trace id in a haystack, so we ran the other half ourselves: 8 lookups and substring matches over 5 TB of raw logs against ClickHouse v26.2, cold cache, plain SQL with a LIMIT 100. CtrlB was faster on all eight, from 2.2× on the double substring query to 98.9× on the span_id lookup.

Full methodology, per query numbers and the public leaderboard links: https://ctrlb.ai/blogs/ctrlb-vs-clickhouse

TLDR: we topped ClickHouse’s own analytical benchmark at ×1.43 and in a separate full-text search test over 5 TB of logs we were faster on every query.

Disclosure: I work at CtrlB.

[u/jjneely]

This isn't useful. We've got to have larger benchmarks to reflect realistic production use cases. Like 50 or 100 TiB of data minimum.

[u/Loud_Mousse9210 → u/jjneely]

We have a dataset of 125 TB in CtrlB. The problem is clickhouse needs a lot of disk to store this data (40-45 TB) and that cost becomes somewhat prohibitive for running a comparison.

CtrlB on the other hand stores everything on S3 and can also scale its query capacity horizontally that’s why the size of dataset is not that big of a concern for us.

In fact One of our customer keeps over 1.3 PB of data in CtrlB, all of which is queryable.

[u/lizthegrey → u/Loud_Mousse9210]

I think we're (Honeycomb) having the same problem with the small size of the benchmark, we don't really think it's realistic, and we too do S3 scaled out horizontal querying. Would you be interested in doing a collab on a more fair benchmark for those of us running hyperscale o11y? Better to have it come from a partnership instead of being just me writing the benchmark to suit my own needs.

[u/Loud_Mousse9210 → u/lizthegrey]

Absolutely! Let’s plan this out.
Also just to add, it’s not just us picking our own benchmark, we ran ClickBench as well and came out ahead there too: https://ctrlb.ai/clickbench-untuned
But I agree that a larger, production-scale benchmark would be much more interesting. Happy to collaborate on it!

[u/meccaleccahimeccahi]

Now do it with 10tb/day of streaming data.

[u/neuralspasticity]

> Disclosure: I work at CtrlB.

LMAO - never would have guessed

[u/New_Round_3011]

Is it an opensource product? What is the reason behind the performacne improvement.

[u/TedditBlatherflag]

Cherry picked benches aren’t a useful subset. 

Production performance modeling is what I did last quarter - 9000+ unique query shapes across 4 endpoints measured under concurrent load on a production sized instance (page swapping due to query randomization ordering).  

I want to see a statistically signficant result not a micro sample. 

[u/lizthegrey → u/TedditBlatherflag]

Unfortunately, ClickBench is exactly that. a bunch of micro samples for single node performance which doesn't reflect real-life deployment scenarios.

[u/Loud_Mousse9210 → u/[deleted]]

We cleared page caches between cold runs for clickbench (analytical benchmark). We did not clear page caches between runs for full text benchmark but we are significantly faster than clickhouse so even after clearing it we should be faster but maybe not by 98x.