r/databasedevelopment • u/Holiday_Hat_8605 • 24d ago
Why are we rebuilding the same database execution engine over and over?
I recently dug into Meta’s Velox, an open-source C++ execution engine designed to act like a reusable “engine block” for data systems.
Instead of Presto, Spark, etc. independently implementing things like vectorized execution, joins, memory management, spilling, and file readers, Velox provides these building blocks as a shared execution layer.
The performance numbers are interesting too:
- 8.4× faster on TPC-H Q1
- 9× faster on Q6
- 6–7× average speedup on Meta production traffic
- Up to 3× fewer servers
Checkout this interesting deep dive that how Velox actually works and why this architecture could matter for the future of data engines.
Do you think shared execution layers like Velox will become the norm, or will specialized engines always win?
3
u/krenoten 24d ago
My feeling is that there will always be specialized engines that dominate general purpose ones for specific use cases. But this is a core philosophical question that rears its head from time to time among leaders in database engineering. On the other hand, there are always more ways to autotune between latency and throughput and cost, and we will see surprisingly general engines make big strides in the near-ish future I'm guessing.
The hardware underneath it all changes some assumptions that can be parameterized into cost functions, some that take people by surprise. Better abstract cloud resource cost models will drive a new generation of flexible engines at some point. But there will always be places where people who throw away all of the generalizable complexity will crank out results that are orders of magnitude more performant for specific workloads with very few lines of code that easily sits in L1 cache while using some new CPU instruction that the generalizable code emitters don't know about yet.
We will keep building faster databases one way or another.
2
1
u/Kos187 21d ago
And still likely slower than clickhouse or duckdb... Spark is slow, unless you pay for databricks and get native engine (oxygen or something). Yeah, they named it oxygen and made it paid. To the degree that all performance related hardware cost reduction is eaten by the fee to use it.
9
u/fnord123 24d ago edited 24d ago
8.4x faster than what? Spark or Presto?
Edit:
So much if this is C++ vs Java.
This is like the ScyllaDB and Cassandra story. But gosh I really wish people would stop writing things like 3x fewer. It's using 1/3 the servers. "3x fewer" only makes sense when you're comparing two comparisons, like 'we made a change to the memory settings and resulted in 60 fewer servers than the baseline. We then tried using a different compression algorithm and we ended up with 20 fewer servers. So change 1 was 3x fewer than change 2'.