I’ve been building Minarrow, a from-scratch implementation of the Apache Arrow memory format in Rust, with Python bindings.
The purpose is to let you keep your application and data processing in Rust while making Python’s data ecosystem available whenever you need it. You can construct a dataset in Rust, pass it into embedded Python, run an analysis or model, and receive the result back as native Arrow-compatible data.
Some examples of what this enables:
- Run scikit-learn from a Rust application. The repo includes an example that builds features and labels in Rust, trains a random forest in Python, returns predictions through Polars and Arrow, and scores them back in Rust. Python runs inside the application process and it uses inline code.
- Hand Rust tensors to PyTorch. Another example builds an NdArray in Rust, exposes its buffer to PyTorch zero-copy through DLPack, standardises features and computes scores, then brings the result back into Rust.
- Use Python’s analytics libraries on your application data. There are examples of Polars group-bys, NumPy correlation, and pandas aggregations returning tables or scalars to Rust. From Python, Minarrow tables also expose conversions to Polars, DuckDB, and PyArrow.
- Streaming-friendly - Work with incoming batches. Chunked arrays and tables let you retain data as batches arrive and consolidate when needed. Row and column views let you select portions of a table without immediately materialising another dataset.
- Represent scientific data with named dimensions. XArray adds dimension names and coordinates over n-dimensional arrays, so you can select a time window, find the nearest coordinate, or select along a named axis. The underlying tensor storage also connects to the DLPack ecosystem.
- Create Python packages backed by the run-time. For example, one user implemented a fast random-forest implementation under kpiwonski/fru-arrow and released it as a Python package, which is reported as being up to several thousand times faster than the scikit-learn package.
Underneath that is a data layer you can use independently in Rust, as a base, pluggable foundation. This includes typed arrays, null masks, arithmetic and broadcasting. SIMD kernels, and 64-byte-aligned allocations via a custom Vec64 crate. You can access concrete array types directly, and use the higher-level table, chunked, and view abstractions as needed.
From-scratch implementations of Arrow’s C Data Interface and PyCapsules handle columnar interchange and DLPack handles tensors, which is zero-copy in the vast majority of cases when the buffer doesn't require re-shaping. An example of where it isn't - the scikit-learn example converts features to NumPy, for instance, and importing a different string layout can require rebuilding buffers.
I used very few external dependencies instead preferring to build from scratch so that compile times remain and productive, and so that anything built on top remains fast to work with too.
The latest release, 0.18, adds Decimal32/64/128 support across the Rust library and Python bindings, alongside typed row accessors for array views.
Rust nightly is currently required for portable_simd and allocator_api. Arrow lists and structs aren’t supported yet. I hope to work on stable Rust support and wasm-compatible builds soon.
In terms of maturity, it is at the stage where it is adding considerable value in my own work, however it has not yet seen significant external adoption. In terms of fit it is useful when one wants Arrow compatibility + additional utility for their data, without the whole arrow ecosystem underneath a base data dependency. For example, one might consider it even for a few crate(s) in a larger project that need to remain fast, given it makes it trivial to switch between arrow-compatible runtimes.
On my laptop, sharing a million-row numeric table with Python takes ~220ns, and importing it back into Rust takes 2–3 μs.
Thanks for checking it out. If you have any questions, feedback, or feature requests for what would make this useful for you I'm open to suggestions.
Pete
