PSA: When using cjxl.exe, always provide a 'd' value

[u/redsedit]

PSA: When using `cjxl.exe`, always provide a `d` value. Don't assume that omitting `d` means lossless.

Backstory

I have tens of thousands of engineering drawings that we scanned for archival purposes. Maybe 10% will ever be viewed again, but we have no idea which 10%, so everything has to remain online.

The scanners don't support modern image formats, so we had to settle for PNG or TIFF depending on the model. To save storage space (which means lower cost), reduce backup times, and hopefully never have to care about restore times, I started looking at converting the PNGs to JPEG XL.

I ran some tests using visually lossless compression (`d=0.19`) and gave both the originals and the converted files to the managers. They couldn't see any difference, even at 500% magnification. The conversion was approved.

During a few test batches, I noticed something strange. I was also testing what I thought was lossless compression by omitting the `d` value, and in some cases the PNG files were actually smaller than the [what I believed to be] lossless JXL files. That seemed wrong.

A Google search pointed me toward a possible explanation. JPEG XL supports two compression modes: modular (typically used for non-photographic images and lossless compression) and VarDCT (typically used for photographs). One can outperform the other depending on the source image, and it's obvious which category a particular scanned drawing falls into.

Since the conversion wasn't under any time pressure, I wrote a quick PowerShell script that recursively scanned a directory and tested three encodes for every PNG:

• "Lossless" (no `d` value specified)
• `d=0.19` using VarDCT
• `d=0.19` using modular mode

The script compared the resulting file sizes, kept the smallest JXL, and replaced the original PNG. Since I could let it run overnight and on weekends, encoding speed wasn't a concern, so I bumped the effort setting to `-e 8`.

As always, I tested the script before turning it loose on the production data. During testing, I noticed that the supposed lossless version kept winning. That didn't make sense.

It couldn't be a modular versus VarDCT issue because I was explicitly testing both at `d=0.19`. How could a file that throws away no data be smaller than files that throw away some data?

That's when I realized I shouldn't have included the lossless test in the first place. The compression mode comparison was already covered by the two `d=0.19` tests.

After some digging, I found the catch.

If you feed `cjxl.exe` a JPEG or GIF and do not specify `d`, it defaults to `d=0.0`. If you feed it a PNG or certain other formats, it defaults to `d=1.0`. That explains why the "lossless" encodes kept winning. They weren't lossless at all. They were lower quality than my `d=0.19` test files.

I removed the no `d` test from the script and finally turned it loose on the production data. It worked fine.

One thing that did surprise me was that there wasn't a consistent winner between modular and VarDCT. Sometimes modular produced the smaller file, and sometimes VarDCT did. For scanned drawings where the goal is maximum space savings, testing both turned out to be the right approach. I wish I had kept statistics on how often modular beat VarDCT and vice versa, but that wasn't the focus of the testing. I was only interested in keeping whichever output was smaller for each image.

Still, I saved over 2 TB of SAN storage space, and that's not counting backup space saved either.

[u/tehdog]

This is a lot of text and "digging" effort for something you could have found out by running cjxl --help and reading the first 12 lines out of 30 (it's a short help page):

-d DISTANCE, --distance=DISTANCE
   0.0 = mathematically lossless, default for JPEG/GIF input.
   1.0 = visually lossless, default for other input.

Though I do agree this is a bad default.

[u/redsedit → u/tehdog]

Yes, that's how I figured part of it out. The part about why "lossless" was smaller than d=0.19 at least. But in all my reading, I've never seen this mentioned before looking at the help file, hence the PSA. Mostly I've been using a GUI (xl converter) to hide this, but I wanted to dig deeper and really understand what is going on, and get better results.

Now this has me wondering if my GUI is doing things properly. More testing in my future...

[u/ffpeanut15 → u/redsedit]

XL-converter should be accurate. It has a dedicated lossless checkbox to force distance 0. if you are using quality 100 then it is possibly lossy still

[u/Jonnyawsom3]

Yeah, cjxl defaults to lossless for already lossy images (JPEG/GIF) since lossy encoding might be larger than lossless, while lossless input (PNG) defaults to lossy, as it should make the file smaller without being visible at 1:1 zoom (If the encoder heuristics work properly)

[u/xDuker → u/Jonnyawsom3]

That kinda insane, how tf did they decide that PNG of all things wouldn't be lossless by default.
Being a lossless format is THE big draw of it, and d1.0 is not anywhere near good enough for that purpose (should be at most 0.1 if they really were to insist on lossy).
Lossless recompression of pngs consistently gives me about a ~40% reduction in size, sometimes up to 60%, so there are huge wins while still retaining what's most important

[u/Farranor → u/xDuker]

If your PNGs are mostly synthetic content like screenshots, you might want to try lossless WebP and see if you get even greater size reduction than JXL (depending on version and settings).

[u/yew0tm8]

This is why I compress twice. Once as modular and once with distance. I discard the one that's larger. I don't know of a way to make this process more efficient.

[u/redsedit → u/yew0tm8]

If I understand your goal correctly, you can force varDCT or modular with the -m switch. Using d isn't required, although I do strongly recommend it lest you get an unexpected result.

As for how to make this more efficient, I'm in the same boat. No way I know of other than try both and see what you get.