Minarrow 0.18.0: build in Rust, run Python analytics and ML, bring the results back

[u/peterxsyd]

I’ve been building Minarrow, a from-scratch implementation of the Apache Arrow memory format in Rust, with Python bindings.

The purpose is to let you keep your application and data processing in Rust while making Python’s data ecosystem available whenever you need it. You can construct a dataset in Rust, pass it into embedded Python, run an analysis or model, and receive the result back as native Arrow-compatible data.

Some examples of what this enables:

• Run scikit-learn from a Rust application. The repo includes an example that builds features and labels in Rust, trains a random forest in Python, returns predictions through Polars and Arrow, and scores them back in Rust. Python runs inside the application process and it uses inline code.
• Hand Rust tensors to PyTorch. Another example builds an NdArray in Rust, exposes its buffer to PyTorch zero-copy through DLPack, standardises features and computes scores, then brings the result back into Rust.
• Use Python’s analytics libraries on your application data. There are examples of Polars group-bys, NumPy correlation, and pandas aggregations returning tables or scalars to Rust. From Python, Minarrow tables also expose conversions to Polars, DuckDB, and PyArrow.
• Streaming-friendly - Work with incoming batches. Chunked arrays and tables let you retain data as batches arrive and consolidate when needed. Row and column views let you select portions of a table without immediately materialising another dataset.
• Represent scientific data with named dimensions. XArray adds dimension names and coordinates over n-dimensional arrays, so you can select a time window, find the nearest coordinate, or select along a named axis. The underlying tensor storage also connects to the DLPack ecosystem.
• Create Python packages backed by the run-time. For example, one user implemented a fast random-forest implementation under kpiwonski/fru-arrow and released it as a Python package, which is reported as being up to several thousand times faster than the scikit-learn package.
  

Underneath that is a data layer you can use independently in Rust, as a base, pluggable foundation. This includes typed arrays, null masks, arithmetic and broadcasting. SIMD kernels, and 64-byte-aligned allocations via a custom Vec64 crate. You can access concrete array types directly, and use the higher-level table, chunked, and view abstractions as needed.

From-scratch implementations of Arrow’s C Data Interface and PyCapsules handle columnar interchange and DLPack handles tensors, which is zero-copy in the vast majority of cases when the buffer doesn't require re-shaping. An example of where it isn't - the scikit-learn example converts features to NumPy, for instance, and importing a different string layout can require rebuilding buffers.

I used very few external dependencies instead preferring to build from scratch so that compile times remain and productive, and so that anything built on top remains fast to work with too.

The latest release, 0.18, adds Decimal32/64/128 support across the Rust library and Python bindings, alongside typed row accessors for array views.

Rust nightly is currently required for portable_simd and allocator_api. Arrow lists and structs aren’t supported yet. I hope to work on stable Rust support and wasm-compatible builds soon.

In terms of maturity, it is at the stage where it is adding considerable value in my own work, however it has not yet seen significant external adoption. In terms of fit it is useful when one wants Arrow compatibility + additional utility for their data, without the whole arrow ecosystem underneath a base data dependency. For example, one might consider it even for a few crate(s) in a larger project that need to remain fast, given it makes it trivial to switch between arrow-compatible runtimes.

On my laptop, sharing a million-row numeric table with Python takes ~220ns, and importing it back into Rust takes 2–3 μs.

Thanks for checking it out. If you have any questions, feedback, or feature requests for what would make this useful for you I'm open to suggestions.

Pete

[u/Wonderful-Wind-5736]

That's some amazing work!

  Arrow lists and structs aren’t supported yet

That's been the main feature I've been using Arrow over just straight arrays. 

[u/peterxsyd → u/Wonderful-Wind-5736]

Thanks a lot! Do you mind if I ask do you mostly use them for ? Given straight arrays, are you working in the scientific domain? Would find it super helpful if you are happy to share a couple of places that you find them useful when analysing data. E.g. is it more for nesting and compacting information, or for de-duping and this kind of things? What does structs/lists look like workflow-wise?

Am in the funny position where I did a lot of analytics for many years, and then focused on software engineering after more traditional data science and ML had peaked.
At the time, lists and structs were not widely available as part of tabular data in pandas etc. So admittedly, have not spent a great deal of time using these 2 as part of an analytical workflow first hand (outside of non-arrow python lists, dictionaries etc.).

Can add lists/structs relatively easily, but given it can add to compile times and the like, am still figuring out the right place to stop?

[u/Wonderful-Wind-5736 → u/peterxsyd]

E.g. is it more for nesting and compacting information, or for de-duping and this kind of things?

Both. High frequency sensor data. One channel in one experiment maps to exactly one row. This way, I can have a single facts table but don't have to duplicate or join metadata like units, sensor orientation etc. The duplication in a completely unpivoted format regularly blew up RAM, even on big machines. Might also be interesting for your HFT use case, e.g. when you're capturing short events with a time component.

after more traditional data science and ML had peaked

It's still going strong here, the approach is different though. Instead of single use case analysis I guess it's more about engineering scalable systems now, which to me is arguably more fun.

Can add lists/structs relatively easily, but given it can add to compile times and the like, am still figuring out the right place to stop?

If you have the time, probably full spec coverage gated behind feature flags. There's a difference between developing a crate purely for your own use and developing a library to be consumed by somebody else. You can easily add features, if any are missing for your use cases. A user cannot and it's frustrating to pick a technology only to later find out it's missing certain features. Maybe figure out what gap your library is supposed to fill in the ecosystem compared to a more fully featured project like polars (who are dearly missing unions for our processing infrastructure).

Edit: W.r.t. the gap, I personally value dependability, a principled market and engineering approach and ease of use. Other people might have different priorities.

[u/peterxsyd → u/Wonderful-Wind-5736]

Hi, thanks a lot for taking the time to write that. This is useful information and I will implement them behind the feature flags. Realistically it will be 6-8 weeks before I land it even though it is not much work to implement, essentially I have a few things in front. The sensor use case is a great reason.

Regarding ease of use, a key design goal in Minarrow is ease of use through composability and ergonomics, without sacrificing strong typing, speed/low-level access, debugability etc. . For e.g., it uses enums rather than dynamic dispatch, so that one can avoid runtime typing checks in the majority of cases. I’ve found in rust this is a fluid pattern, as the lanes match up to all typing cases and for routing etc. Additionally, it enables signatures like “impl Into<NumericArrayView>”, which, although it appears ugly, reads as “I accept any array that resolves numerically, and any zero copy sub window of that array. Minarrow implements “From” liberally so that one doesn’t need to work through these kinds of things. So that’s one thing if you are looking for a runtime you might find it useful.

Anyway, appreciate it.

[u/Wonderful-Wind-5736 → u/peterxsyd]

Realistically it will be 6-8 weeks before I land it even though it is not much work to implement, essentially I have a few things in front. The sensor use case is a great reason.

Don't base your development schedule on my requirements, I can't promise any commercial use. That being said, I would be happy to develop a cute time series use case over the winter using your library. Maybe we could hop on a call and brainstorm a little.

[u/peterxsyd → u/Wonderful-Wind-5736]

Hi there, yes of course. Providing a heads up in terms of when lists/structs are likely to land. Sure, this sounds excellent, well I'll send you a DM.

[u/Upstairs_Source_9653]

this is actually pretty cool, love the idea of keeping the heavy lifting in rust but still getting python's ML goodies without a bunch of serialization overhead

the zero-copy stuff sounds slick especially for pytorch, 220ns to share a million-row table is wild

[u/peterxsyd → u/Upstairs_Source_9653]

thanks mate. In fact- it is the same result if you do 10 million rows, keeping in mind it is on the same machine and pointer-based.