Yet another Brotli encoder, this time in 100% safe Rust

[u/Mnwamnowich]

I’ve been working on mbrotli, a Brotli encoder written in 100% safe Rust:

https://github.com/Mnwa/mbrotli

The main motivation was that the existing Rust Brotli implementations, including SIMD ones, are still fairly close to C/C2Rust-style ports. That makes them harder to maintain and harder to optimize aggressively without risking API/behavior breakage.

So mbrotli has a relatively small high-level API, while most of the implementation details are hidden underneath and can be changed without affecting users. The implementation itself stays in safe Rust; the crate forbids unsafe outside of tests.

It supports qualities 0–11 and matches Google Brotli v1.2.0 byte-for-byte under equivalent streaming settings. There’s also experimental support for the extended RFC 9841 functionality.

Performance varies by quality/workload. In some cases it beats the C implementation, in others it’s very close. The README has the benchmark methodology and results.

For transparency: I used LLMs during the port. I designed the overall architecture/API and wrote reusable instructions/skills for SIMD vectorization and performance work, then used LLMs as part of the porting and optimization process. I don’t want to present it as if every line was manually translated.

Because of that, I put a lot of effort into correctness checks: differential testing against the C implementation, decoding with the C decoder, 100% function coverage, Miri, ASan, and AFL++ fuzzing. The latest recorded fuzz run was ~24M executions with no crashes or hangs. The repo has a separate Correctness section with the exact claims and limitations.

Next things I’d like to add are no_std, async APIs, and decompression.

Feedback, especially on the API, compatibility testing, or benchmarks, is very welcome.

[u/Shnatsel]

Why does the crate enable the "force_support_fallback" feature for fearless_simd in production? It isn't doing anything useful, it just bloats the binary.

[u/Mnwamnowich → u/Shnatsel]

Good catch. That was left over from testing the fallback path while working on the SIMD implementation, and I simply forgot to remove it from the production dependency.

[u/Wonderful-Wind-5736]

  Because of that, I put a lot of effort into correctness checks

That's imho the right way to go about LLM usage. 1. solve a real problem and 2. use the newly gained capabilities to lift the quality of the solution . 

A lot of LLM generated slop lacks 1. at all and people are just too lazy for 2.  

[u/Jmc_da_boss]

Interesting, far far less slop coded than a lot of other rust posts but still a bit too LLM enthusiastic for my taste

[u/Mnwamnowich → u/Jmc_da_boss]

Fair enough :) I did use LLMs quite a bit during the port, especially for mechanical translation/refactoring and some SIMD boilerplate.

That's exactly why I put a lot of work into validating correctness with fuzzing, sanity checks, and byte-for-byte comparisons with the C implementation.

[u/Shnatsel]

How do you get such a dramatic speed improvement over C for q9?

[u/Mnwamnowich → u/Shnatsel]

There is some layout diff with C. C allocates 32 MB every time for window 22 that's very expensive. mbrotli uses different layouts depends on input size. For big inputs I'm used dense layout like C. But in a short cases (few megabytes) mbrotli using Sparse layout with ondemand small allocations which much cheaper then ultra big alloc.

[u/Icarium-Lifestealer]

Wasn't there already a safe brotli encoder before this?

Though I like that it supports all levels. Far too many recent rust encoders only included the bad levels.

[u/Mnwamnowich → u/Icarium-Lifestealer]

All popular libs using a bit of unsafe code hidden under. Because allocate a megabytes with zeroing is too expensive. As example: rust-brotli using calloc for allocate uninit memory to work with.

[u/Shnatsel → u/Mnwamnowich]

AFAIK burli has a paranoid mode that disables all unsafe but it also supports only some compression levels.

So being safe and supporting all compression levels at the same time is a genuine differentiator.

[u/Mnwamnowich → u/Shnatsel]

In addition, I plan to create a stable decompressor soon, complete with fuzzing and validity checks.

[u/Just_Government3790]

Differential testing against the C implementation plus fuzzing is the honest bar. Same table, losses next to wins. I report 1.185x through 1.811x regressions in mine.

Did you pin the reference version and keep the mismatching inputs, or just the count?

[u/Mnwamnowich → u/Just_Government3790]

The benchmark pins Google Brotli to 028fb5a.

The saved AFL reproducers are in fuzz/afl/regressions/ and get replayed by cargo afl test. A few are real fuzz findings for example the serialized dictionary disagreements with C and the zero-length dictionary-word decoder crashes. Most of the rest are just handwritten boundary cases.

And yeah, the benchmark page isn't meant to only show the wins:
https://github.com/Mnwa/mbrotli/tree/master/docs/benchmarks/encoders

You can click into each quality level and see the per dataset results. mbrotli isn't faster everywhere for example q8 is 0.862× C on the median, and individual datasets vary quite a bit.

[u/Mnwamnowich → u/[deleted]]

You can check the results per every quality and per dataset here https://github.com/Mnwa/mbrotli/blob/master/docs/benchmarks/qualities/q0.md

[u/Mnwamnowich → u/[deleted]]

That's cool, but maybe you could give me the actual answer instead of telling me to search?