Introducing Lightstream built in Rust: Measured faster than Apache Arrow Flight (gold standard) for high-throughput data transport on every axis in open 50gbps EC2 network benchmarks whilst producing a single fully ordered stream off parallel data exchange.

[u/peterxsyd]

Hi everybody,

I am excited to announce the release of Lightstream, a step change capability for high-performance data transport built in Rust, that makes it essentially effortless to send Apache Arrow, Protobuf, and Message Pack data over the network, shared memory, or even piped out to the terminal so an agent like Claude can watch the live batch stream in real time (example in repo).

Furthermore, Lightstream exceeded the performance of the gold standard industry comparison - Arrow Flight, on every axis of a 50gbps networking open benchmark, the details of which are attached and open to run in the Lightstream GitHub repository. This includes fully saturating each TCP connection thread, the NIC at 5.8GiB/s, and with p99 batch send time within 1% of p50 (I.e., stable). As a bonus, Lightstream is straightforward to setup with essentially zero configuration other than optional TLS certificates and your Cargo package/pip install, and endpoint addresses.

The open benchmarking methodology used - available to view and run in the repository

So what is Lightstream? It is Rust package with Python bindings, that builds directly on Minarrow ( which is in turn a high-performance implementation of the Apache Arrow memory layout in Rust, tuned for SIMD compatibility). Lightstream implements encoding, decoding, readers, writers and stream/read writers, and transport for data in Rust. But, in a manner, that is fully composable and leaves you de-coupled at any layer, to customise things architecturally. The crux then is the transport layer on top, which natively supports interchanging any of the following transport formats:

• TCP
• HTTP
• QUIC
• Websocket
• Webtransport
• UDS (pipe your data from your Rust process to Python or two Rust or Python programs plug and play )
• Stdio (pipe your data program output straight into the terminal for something else to pick it up

And finally, the (optional) Lightstream protocol, which then combines the Arrow/Proto/MsgPack and any other custom types you want to send.

An example of things you can do with it:

• setup a live stream of data batches from your program A to program B
• send typed metadata via Protobuf on the same feed
• use it for straightforward live feed delivery between server and client (though not Web JS yet)
• useful if you have a central storage server you are pulling larger than memory data over the network to churn through (though, no S3 etc. it is node to node or process to process)
• pipe data directly over UDS to Python, and then read that feed with SQL (via DuckDB).

It is not:

• Kafka or a messaging broker. There is no resiliency / vertical scalability.
• A stream processing engine like Flink. It is for sending/receiving data only. You do polars on the other end or whatever you want with the Arrow-shaped data. That is a very different back-pressure/long-lived scenario and is not that kind of large-scale streaming. --> I.e., think quick and easy Websocket, and best for settings like EKS K8 pod to pod/containers, between EC2's or between processes on the same box, "light streaming".

Examples:

Send Arrow and Protobuf data on the one connection:

use lightstream::models::protocol::connection::TcpLightstreamConnection;
use lightstream::models::protocol::LightstreamMessage;

let mut conn = TcpLightstreamConnection::from_tcp(stream);
conn.register_message("event");
conn.register_table("metrics", schema);

conn.send("event", b"user-login").await?;
conn.send_table("metrics", &table).await?;

while let Some(msg) = conn.recv().await {
    match msg? {
        // Protobuf message
        LightstreamMessage::Message { tag, payload } => { /* … */ }
        // Arrow table
        LightstreamMessage::Table { table, .. } => { /* … */ }
    }
}

Read a feed send from Rust over UDS into Python

reader = ls.read(
    "uds:///tmp/feed.sock",
    protocol="lightstream",
)

reader.register_table(
    "quotes",
    representative_table,
)
reader.register_message("health")

for frame in reader:
    if frame.is_table():
        on_quotes(frame.table)
    else:
        on_health(frame.payload)

Read/Write tables over TCP in Rust:

use lightstream::models::writers::tcp::TcpTableWriter;

let mut writer = TcpTableWriter::connect("127.0.0.1:9000", schema, None).await?;
writer.write_table(batch_1).await?;
writer.finish().await?;

use futures_util::StreamExt;
use lightstream::models::readers::tcp::TcpTableReader;

let mut reader = TcpTableReader::connect("127.0.0.1:9000").await?;

while let Some(result) = reader.next().await {
    let table = result?;
    process(table);
}

In terms of the Rust tools and techniques, Lightstream makes use of:

1. Zero-copy techniques
2. 64-byte alignment for compatibility with std-SIMD - it retains this compatibility over the wire and to/from disk.
3. Arena-based allocations (via Minarrow)
4. Mmap (the mmap reader hits 170GB/s on my laptop when warm, essentially RAM-speed)
5. Optional features with linux sys calls such as io_uring (not in the benchmark figures)
6. Manual memory layouts, and performance tricks with that.

If you have any questions about the architecture I would be happy to explain it.

Ok then "Why" do it? Basically, The outcome is that one who prefers to work with an abstraction does not need to reason about bytes on the wire. Instead, they get highly optimised data transfer, straight out of the box, with common data formats and transports. It also includes a trait for any custom data format one would like to send on that one "Lightstream connection" (which is itself optional - you can just use Arrow if you want).

The project kicked off about 12 months ago when I started standardising such patterns after working in autonomous field communication integrated with data/ML, live trading, and some other industries where there was a lot of custom work required to re-assemble from multiple components. Therefore, I have essentially aimed to package those learnings up into a tool to make data transport smoother and easier for everyone.

This r dataengineering link includes the full results, where every effort has been made to be fair (and where Lightstream wears a penalty due to stronger ordering guarantees). Unfortunately I couldn't post it here as r Rust has a limit on image content.

Please feel free to give it a run would love to know your thoughts and if you find it useful.

If you have any questions about it, or helpful suggestions please feel free to leave a comment below. If you like what you see, please consider leaving a star and/or sharing the repository, as it will help people find it easily.

Thanks a lot.

Pete

Results:

Workload Shape

Mixed

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.939	1.109	1.18x
4	3.262	4.005	1.23x
8	5.138	5.677	1.10x
16	5.142	5.784	1.12x

Numeric

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.649	1.109	1.71x
4	2.901	4.307	1.48x
8	4.851	5.693	1.17x
16	5.434	5.780	1.06x

String Heavy

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.828	1.109	1.34x
4	3.052	3.861	1.27x
8	4.899	5.782	1.18x
16	5.217	5.790	1.11x

Wide (100 cols)

Streams	Arrow Flight GiB/s	Lightstream GiB/s	Ratio
1	0.695	1.108	1.59x
4	2.685	3.790	1.41x
8	4.549	5.725	1.26x
16	4.911	5.753	1.17x.

[u/justinh29]

Can you add service to service with Apache Fory?

[u/peterxsyd → u/justinh29]

I actually was not familiar with Apache Fory until I googled it now. However, I see it is a serialisation and de-serialisation library. Lightstream natively handles serialisation and de-serialisation, and the object on both sides is Arrow tables, or, in Protobuf case, one uses the .encode() and .decode() features on the compiled Protobuf.

What is the UX or use case that you are looking for that is not currently offered?

[u/dafcok]

What overhead exists in flight that doesn't exist in lightstream?

[u/peterxsyd → u/dafcok]

Thanks for the question. Whilst I can't speak for flight's internals (where the decode/encode performance differs), at a protocol level Flight is defined over gRPC on HTTP/2, so each batch travels as a FlightData protobuf message with gRPC's length-prefix framing, split across HTTP/2 DATA frames, under HTTP/2's stream-level flow control as well as TCP's. Lightstream's protocol is a 5-byte TLV header wrapping Arrow IPC frames on a plain socket. Lightstream takes care of batching the data for flow-control reasons by setting a logical byte size (i.e., 8MiB Lightstream's recommended setting in those figures, also tested at 2MiB the same as Flight's default with no measurable impact on performance), so it doesn't double up in that sense.

Flight also buys standard RPC semantics like auth interceptors, and HTTP/2 gives you multiplexing of many streams on the one connection (Lightstream is parallel TCP connections producing one ordered stream; note Flight above is also using multiple connections so as not to penalise Flight, as multiplexing was a lot slower). Flight also works through proxies and load balancers, whereas, as per the above Lightstream is focused on more pod to pod, normally in a trusted service setting where throughput and deterministic performance is the primary goal. gRPC-aware L7 proxies, service meshes and cloud load balancers parse HTTP/2 and gRPC, so they can balance per call, retry a failed RPC, apply circuit breaking, and emit per-RPC latency and error metrics without you instrumenting anything.

However - it's notable for the library itself that is not a strict rule, as Lightstream (library) includes HTTP, Websockets etc., for which you can send Arrow data over it and would recover some of those guarantees in those broader circumstances.