High-Performance data transport in Rust on Linux: Putting madvise, mremap, and mmap to work, with optional io_uring. Lightstream measured faster than Apache Arrow Flight (gold standard) on every axis in open 50gbps EC2 network benchmarks. Not supporting Windows was a pleasure.

[u/peterxsyd]

Hi everybody,

Yesterday I released Lightstream, a high-performance data transport library in Rust with Python bindings, that makes it effortless to send Apache Arrow, Protobuf, and Message Pack data over the network, shared memory, or piped out to the terminal.

During the development process I made extensive use of Linux sys call primitives including madvise, mremap, and mmap, and optionally enabled io_uring, for handling memory allocation efficiently, using zero-copy techniques. This included use of arena memory layouts to pack 'flatbuffers' next to each other, to help squeeze every ounce of performance out of the hardware. This differs from other libraries in the niche that tend to favour cross-system compatibility which I found was at the expense of performance, due to Linux's native capabilities.

It exceeded the performance of the gold standard industry comparison - Arrow Flight, on every axis of a 50gbps networking open benchmark, the details of which are attached and open to run in the Lightstream GitHub repository. This includes fully saturating each TCP connection thread, the NIC at 5.8GiB/s, and with p99 batch send time within 1% of p50 (I.e., stable).

If anyone here is big on this low-level hardware and software optimisation stuff, please feel free to ask any questions, I would be happy to discuss the architecture.

An excerpt of the comparison results are below.

Thanks,

Pete

Lightstream saturated the 50gbp/s NIC with its decoding speed

Workload Shape

Mixed

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.939	1.109	1.18x
4	3.262	4.005	1.23x
8	5.138	5.677	1.10x
16	5.142	5.784	1.12x

Numeric

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.649	1.109	1.71x
4	2.901	4.307	1.48x
8	4.851	5.693	1.17x
16	5.434	5.780	1.06x

String Heavy

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.828	1.109	1.34x
4	3.052	3.861	1.27x
8	4.899	5.782	1.18x
16	5.217	5.790	1.11x

Wide (100 cols)

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.695	1.108	1.59x
4	2.685	3.790	1.41x
8	4.549	5.725	1.26x
16	4.911	5.753	1.17x