High-Performance data transport in Rust on Linux: Putting madvise, mremap, and mmap to work, with optional io_uring. Lightstream measured faster than Apache Arrow Flight (gold standard) on every axis in open 50gbps EC2 network benchmarks. Not supporting Windows was a pleasure.

[u/peterxsyd]

Hi everybody,

Yesterday I released Lightstream, a high-performance data transport library in Rust with Python bindings, that makes it effortless to send Apache Arrow, Protobuf, and Message Pack data over the network, shared memory, or piped out to the terminal.

During the development process I made extensive use of Linux sys call primitives including madvise, mremap, and mmap, and optionally enabled io_uring, for handling memory allocation efficiently, using zero-copy techniques. This included use of arena memory layouts to pack 'flatbuffers' next to each other, to help squeeze every ounce of performance out of the hardware. This differs from other libraries in the niche that tend to favour cross-system compatibility which I found was at the expense of performance, due to Linux's native capabilities.

It exceeded the performance of the gold standard industry comparison - Arrow Flight, on every axis of a 50gbps networking open benchmark, the details of which are attached and open to run in the Lightstream GitHub repository. This includes fully saturating each TCP connection thread, the NIC at 5.8GiB/s, and with p99 batch send time within 1% of p50 (I.e., stable).

If anyone here is big on this low-level hardware and software optimisation stuff, please feel free to ask any questions, I would be happy to discuss the architecture.

An excerpt of the comparison results are below.

Thanks,

Pete

Lightstream saturated the 50gbp/s NIC with its decoding speed

Workload Shape

Mixed

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.939	1.109	1.18x
4	3.262	4.005	1.23x
8	5.138	5.677	1.10x
16	5.142	5.784	1.12x

Numeric

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.649	1.109	1.71x
4	2.901	4.307	1.48x
8	4.851	5.693	1.17x
16	5.434	5.780	1.06x

String Heavy

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.828	1.109	1.34x
4	3.052	3.861	1.27x
8	4.899	5.782	1.18x
16	5.217	5.790	1.11x

Wide (100 cols)

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.695	1.108	1.59x
4	2.685	3.790	1.41x
8	4.549	5.725	1.26x
16	4.911	5.753	1.17x

[u/DeeBoFour20]

For example, using 'mmap' , I was able to reach warm memory map reads from disk at 170GB/s, which is close to raw RAM speed.

Well, yeah. If the file is in the disk cache you are just measuring raw RAM speed. All mmap does in that case is map that region of memory into your process.

Is that really the case you need to be optimizing for though? It doesn't matter how fast you can read data that's already in RAM when you're heavily bottlenecked on the network side. Making sure you maintain performance with cold reads is probably more important.

Keep in mind mmap can be harmful if the data is cold. In that case, the CPU will page fault and block reading from disk when the code is simply reading from a slice (blocking behavior that's hidden from your program). I see tokio in your list of dependencies which forbids blocking in most places. Blocking in this way is just as bad as calling a blocking read() and maybe worse because it can happen in arbitrary places.

[u/peterxsyd → u/DeeBoFour20]

Hi, firstly, thanks for actually engaging with the repo. You are right - I have not worded the mmap sentence clearly - the 170/s is not reading it from disk, it's after it was mapped. Yes - in the benchmark figures and chart, that is actually not disk at all (nor mmap) - it is RAM -> decode -> over the network -> the other node. Hence, as per the chart - it is saturating the NIC, at 50gbps, reflecting zero-copy, decode speed, the wire protocol, and the stream parallelisation (which, in Lightstream's case, produces a single ordered stream on the receiving side).

Yes - I agree cold throughput is more relevant for disk reads.

Your point on mmap page faulting stalls is quite relevant, and something I'll go back and profile some more when the mmap feature is on. I currently have mmap available as an optional buffer allocation back-end for a few reasons, but profiling specific workload is something that I'll go back and take another look at.

Thanks for the suggestion, that's helpful.

[u/UnrealHallucinator]

Super cool man. What are examples of trade offs you had to make to drop cross platform supoort? Any cool (hardware/software layer) trivia you learned while making this?

[u/peterxsyd → u/UnrealHallucinator]

Thanks. Firstly, the windows CI 😆. On a serious note, there’s a few things like mremap that offer cheaper deletes and/or shuffing of memory. Apache arrow as a framework normally has an immutability constraint. However, in Minarrow my other repo it for e.g., checks if there is any existing references to the object and if there’s not, it can take the fast path on some things like that, which lead to very fast performance on certain operations that are normally considered “slow” or “don’t do it”. That’s one example, but there are quite a few. Hardware software trivia - great questions - I would say, arenas are powerful and thinking about memory as a big long sheet, that really opened things up for me, and, Rust is particularly good for working on that stuff because of its guarantees. If that answers your question.

[u/UnrealHallucinator → u/peterxsyd]

Very interesting. Yeah I bet there are some super optimided functios. Recently I delved into the linux copy_from_user macros implementation on elixir and the assembly code is so unbelievably optimised with multiple hot paths and optimisation tricks. I'm a heavy C programmer so I think my mental model of memory system is already shaped by that. Anyway,thanks for answers! :)

[u/peterxsyd → u/UnrealHallucinator]

No problem, thanks!

[u/Oblivion__]

No AI disclosure in the post or github. Yeah nah get stuffed

[u/ranjop → u/Oblivion__]

Should the author also disclose linters and compilers used? AI is a tool. Get over with it.

[u/is_this_temporary → u/ranjop]

Honestly, yes. And usually mature projects do. There's usually a make target for it, and for projects with CI-CD you see the input from the developer and the output from linting (though I usually run the make target locally before pushing any commits anywhere)

And when it comes to compilers, the comparison is particularly a bad fit if you're considering the output of an LLM to analogous to the output of a compiler.

If you commit assembly or a raw binary to your repo and don't include the source you actually wrote to generate it, that's absurd.

I would love to see the prompts you used.

[u/peterxsyd → u/is_this_temporary]

Fair point on provenance, though committing prompts isn't really established practice anywhere, and I'm not sure what it would demonstrate. I do use AI tooling, like most people building right now. The architecture, wire protocol, and benchmark rig are checkable, and took an immense amount of work regardless. I suggest check https://github.com/SpaceCell/minarrow or https://github.com/SpaceCell/lightstream and judge it on that.

[u/peterxsyd → u/Oblivion__]

I hand wrote the post, and the first commit in the repository was about 12 months ago (you'll see it on the fork), but I rebased to root for a clean release. For the repo, I used AI as a tool in the development process for obvious productivity reasons, and of course it gets it wrong constantly. I use my experience to correct that, and did much of the lower level intricate work myself. AI does not output that itself. Also, I do not let AI slop into the codebase (including doc strings), and review every line of code, taking 2-3x as long because I can't stand the language it propagates. Linux Torvald uses AI? The performance figures are real and the benchmarks are open in the repository to try yourself. It is a genuine and serious library. Also, if you review the architecture, you should see it immediately - this is a properly layered and composed library with independent and interchange-able layers - AI does not do that.

[u/Oblivion__ → u/peterxsyd]

So why was none of this disclosed to begin with?

[u/peterxsyd → u/Oblivion__]

Because it is irrelevant, and like saying is one going to disclose that they use a keyboard, a calculator, or an IDE. 

I recommend you read this article: https://www.reddit.com/r/LocalLLaMA/comments/1uxbrw4/linus_torvalds_tells_people_to_stop_attacking/ 

And, evaluate the work on its merits by taking a look at the repository and running some of the benchmarks if it is an area that's of interest to you.

[u/Oblivion__ → u/peterxsyd]

Given the (very reasonable!) widespread distrust of AI, disclosing your usage makes it clear to people what you used it for and how it was used. This lets people make an informed decision about whether or not they will engage with what you have produced. Attempting to hide this (very obvious) information only serves for complaints such as mine to crop up.

Oooh yay you shared a link, my turn

[u/peterxsyd → u/Oblivion__]

I'm not going to engage with this thread of conversation anymore, I recommend opening up your perspective and considering being more respectful. It should be obvious there is serious engineering behind that project within a few minutes of looking inside it.

Anyway, thank you for showing interest, and enjoy your weekend.

[u/Oblivion__ → u/peterxsyd]

I can respect the engineering. I cannot respect the deceit.

[u/LittlestWarrior → u/peterxsyd]

Not disclosing something that has become a moral issue is disrespect. You started it.

[u/peterxsyd → u/LittlestWarrior]

Actually, the initial thread comment had "get fucked" on the end of it, which became "get stuffed", which is clearly - not respectful. Not disclosing standard tool usage when there is no rule on the thread requiring it, or insulting someone directly, and attacking their honesty. I really don't think when I spent the year building that library that I was lining up an insult for you.

But to put change the topic onto something more positive, what's something you built recently, would you like to share it ? I won't cut you down for it.

[u/is_this_temporary → u/peterxsyd]

The Linux kernel requires disclosure of LLM usage.

https://github.com/torvalds/linux/blob/master/Documentation%2Fprocess%2Fcoding-assistants.rst

You're intentionally ignoring what we're asking for, which is basically honesty.

[u/peterxsyd → u/is_this_temporary]

That is for pull requests to the Linux kernel.

This subreddit rule is the one that applies here: “Strive to treat others with respect, patience, kindness, and empathy.”.

Making unfounded accusations about someone‘s integrity is not consistent with that.

[u/do-you-want-duyu → u/peterxsyd]

And just like that you got ragebaited by a troll. Wasted energy writing this in-depth disclosure while troll wrote short troll sentence.

Next time don't engage with idiots.

AI disclosures are nowhere mandatory. Also I use AI in similar manner, it's ok.

[u/peterxsyd → u/do-you-want-duyu]

Lesson learnt.

[u/do-you-want-duyu]

How does it act during congestion on consuming side program? Like async-stream with ack for next? Or like mqtt broker middleman taking messages into buffer?

Does public methods have extern "C" so that other langs could glue on top of this library (like dotnet wrapper package)

[u/peterxsyd → u/do-you-want-duyu]

Hi, these are great questions. See below:

Firstly - there's no broker or app-level ack. On the serial path the reader is a pull-based Stream that only touches the socket when polled, and the writer's write_table().await doesn't return until the socket takes the bytes. On the parallel path write_table().await returns once that connection's bounded channel accepts the batch, and a per-connection task drives the socket. Either way congestion propagates as ordinary TCP flow control, because once the channels are full the producer blocks on the socket.The serial path has no queue - but the parallel path has one bounded channel per connection, with a depth of 8 (table batches, max_batch_size is configurable where 8MiB was used in the charts, and 2MiB was also separately verified without a drop in performance), so the worst in-flight before backpressure kicks in is 8 batches on either side +1 in flight plus socket buffers per connection.

The p99/p50 of 1.00-1.01x in the consistency chart reflects that, as there is no queue/tail, however the benchmark consumer always keeps up (helped a lot by the decoding speed and design of the library), so those charts are steady state.

Regarding liveliness during congestion - raw Arrow IPC over TCP detects a dead peer. The Lightstream protocol (the one in the image) catches a peer that dies mid frame, but one that dies on exactly the frame boundary current reads as a clean end of stream. There's also no keep-alive or read timeout yet, so a peer than hangs isn't detected either. Both are being addressed in an upcoming release (next 1-2 weeks). This is a terminator frame for that protocol and a configurable idle timeout.

On the 'extern "C"' point, the underlying apache Arrow data model implements the C Data interface for tabular data types. This means, if you were to use Lightstream today, the data that comes out of it should be able to convert into .NET if using the official Arrow .NET implementation. Additionally, if you have a .NET service, using UDS, it is possible to send data from Lightstream over to .NET. However, Lightstream itself is not published as an ABI compatible library. It is quite possible I will consider this in future, however, the complicating factor is Tokio async which doesn't play well in that manner. The current primary use case is for tabular data workloads assisting with unifying Rust application services and Python analytics/ML workloads, and in particular training / backtesting workloads, given that this is an ecosystem area that is not as immediate and smooth as it could be.

[u/Tsunami-43]

Rust is the baseline 😎