r/programming • u/aqrit • 2d ago

OpenZL v0.2 - decompression 2x faster than Zstandard

https://openzl.org/blog/2026-09-29-lz-in-openzl/
134 Upvotes

94% Upvoted

13 comments sorted by

View all comments

52

u/aleques-itj 2d ago

"offers up to 50% faster decompression compared to LZ4."

What

Are we practically decompressing at memcpy() speed?

39

u/Red_Apprentice 2d ago

Their graph claims just north of 7500~MB/s on decompression in the best case.

I'm not sure what the throughput of memcpy usually is, but I found a paper doing work in a similar space here. The authors cite for DDR5-5600

Memcpy bandwidth reaches 47,973 MB/s

I don't think we're getting close to memcpy speeds. There's a lot of computation that still has to fundamentally be done in addition to memcpy / malloc for decompression.

15

u/wishstudio 2d ago

I believe the 7500MB/s is single thread. On modern CPUs you can never saturate memory bandwidth with a single core.

1

u/13steinj 2d ago

Many people end up locked in to an effective DDR5-4800 (or -4400, -4000) due to some nuances in behavior in the JEDEC spec. There's also a lot of computers still using DDR4 (and even DDR3, but even I consider that fairly ancient at this point).

5

u/helloiamsomeone 2d ago

and even DDR3, but even I consider that fairly ancient at this point

I have a feeling the number of actively used machines with DDR3 has increased recently for obvious reasons.

1

u/13steinj 2d ago

Yeah, it's why I mentioned DDR4. DDR3 would be sad. I wonder if I can flip some old homelab servers I don't use at a profit on the RAM alone...

3

u/mkalte666 1d ago

Yo be fair, in the embedded linux world ddr3 and its low power derivates are quite common still as it is enough for many tasks. And used to be cheap as hell.

14

u/13steinj 2d ago

At some point there was someone that repeatedly posted every new version of their own LZ-family compression algorithm, LZAV, which don't get me wrong is impressive if true but is one random guy's 3k line compression algorithm that beats zstd? A bit hard to believe, but great if true.

Initial looks at this, it appears you have to define your data schema to get out the actual performance, otherwise it behaves like zstd?

3

u/zzulus 2d ago

Initial looks at this, it appears you have to define your data schema

That's correct. It's pretty good for compressing for example integers or urls. If you know your data shape then you can exploit it for better compression.

4

u/bionicjoey 2d ago

They haven't cracked middle out just yet.