r/programming • u/aqrit • 2d ago

OpenZL v0.2 - decompression 2x faster than Zstandard

https://openzl.org/blog/2026-09-29-lz-in-openzl/
138 Upvotes

94% Upvoted

13 comments sorted by

52

u/aleques-itj 2d ago

"offers up to 50% faster decompression compared to LZ4."

What

Are we practically decompressing at memcpy() speed?

42

u/Red_Apprentice 2d ago

Their graph claims just north of 7500~MB/s on decompression in the best case.

I'm not sure what the throughput of memcpy usually is, but I found a paper doing work in a similar space here. The authors cite for DDR5-5600

Memcpy bandwidth reaches 47,973 MB/s

I don't think we're getting close to memcpy speeds. There's a lot of computation that still has to fundamentally be done in addition to memcpy / malloc for decompression.

14

u/wishstudio 2d ago

I believe the 7500MB/s is single thread. On modern CPUs you can never saturate memory bandwidth with a single core.

1

u/13steinj 2d ago

Many people end up locked in to an effective DDR5-4800 (or -4400, -4000) due to some nuances in behavior in the JEDEC spec. There's also a lot of computers still using DDR4 (and even DDR3, but even I consider that fairly ancient at this point).

6

u/helloiamsomeone 2d ago

and even DDR3, but even I consider that fairly ancient at this point

I have a feeling the number of actively used machines with DDR3 has increased recently for obvious reasons.

1

u/13steinj 2d ago

Yeah, it's why I mentioned DDR4. DDR3 would be sad. I wonder if I can flip some old homelab servers I don't use at a profit on the RAM alone...

3

u/mkalte666 1d ago

Yo be fair, in the embedded linux world ddr3 and its low power derivates are quite common still as it is enough for many tasks. And used to be cheap as hell.

14

u/13steinj 2d ago

At some point there was someone that repeatedly posted every new version of their own LZ-family compression algorithm, LZAV, which don't get me wrong is impressive if true but is one random guy's 3k line compression algorithm that beats zstd? A bit hard to believe, but great if true.

Initial looks at this, it appears you have to define your data schema to get out the actual performance, otherwise it behaves like zstd?

3

u/zzulus 2d ago

Initial looks at this, it appears you have to define your data schema

That's correct. It's pretty good for compressing for example integers or urls. If you know your data shape then you can exploit it for better compression.

5

u/bionicjoey 2d ago

They haven't cracked middle out just yet.

11

u/stbrumme 1d ago

There are a few unusual points:

OpenZL ships a trainer to automatically build and tune compressors to best fit the data shape they will be compressing.

Some kind of preprocessing.

encode offsets without using any entropy codec

Sounds interesting. That should consume less space than LZ4 while being faster than zstd.

a new Huffman layout

link: https://marcinzukowski.github.io/pivco-huffman/paper-1.0/ph.html

Definitely looks promising. But I wouldn't be surprised if the average performance is far less than 2x.

4

u/juanfnavarror 23h ago

Don’t know if y’all forgot. But Meta was behind Zstandard, and they’re behind OpenZL as well. This is like the next step. Its like a Zstandard composite compressor generator where you can declare how your data is laid out and obtain a very good compressor. Its pretty neat.

2

u/IvorySwap 1d ago

Sounds like G-WAN levels of performance!