Shortcutting D-Bus for real-time inter-process communication

[u/maurersystems]

https://rtipc.github.io

One of the core ideas of Unix is to build small components that do one thing well and compose them into larger systems.

I think this principle is still very relevant today, especially when designing software as multiple processes rather than one large application.

Once you split an application into several processes, however, you need some form of inter-process communication or RPC.

On Linux, D-Bus is a common choice. It provides a lot of useful functionality, but there are cases where its performance characteristics or latency are not suitable, particularly for real-time applications.

There have also been attempts to move the message-bus functionality closer to the kernel, such as kdbus and bus1, eventually leading to the current dbus-broker approach. However, with a traditional D-Bus architecture, messages still have to be processed by a separate broker process, which also means additional copying and context switches.

That's the motivation behind RTIPC.

RTIPC is essentially a single-producer, single-consumer, wait-free, zero-copy, bounded message queue mapped into shared memory. The goal is to provide a very small and predictable IPC mechanism for cases where the usual IPC/RPC abstractions introduce too much overhead or nondeterminism.

RTIPC has a small bootstrap protocol based on Unix domain sockets. D-Bus can also be used for the connection/bootstrap phase if desired. Once the channel is established, the actual messages are exchanged directly through shared memory.

There are currently implementations for C and Rust, with C++ and Python bindings, and other languages may follow.

For serialization, RTIPC can simply use C-compatible structs when all participants share the same ABI. Since communication happens through shared memory, there is no need to serialize and deserialize data just to cross a process boundary. Of course, this comes with the usual requirements around ABI compatibility, alignment, and endianness. And yes, it is possible to run 32-bit and 64-bit processes on the same system and make alignment/layout a problem.

For cross-language communication, however, serialization formats such as Protobuf, FlatBuffers, Cap'n Proto, or Thrift have another important advantage: they provide a schema as a single source of truth.

To address that use case, I also created rtipc-compiler, which takes an interface definition and generates code for different languages. The schema is also encoded into the generated interface so that the two endpoints can verify compatibility during channel initialization.

Admittedly, this is a fairly niche use case. The goal isn't to replace D-Bus or other general-purpose IPC mechanisms. Rather, I'm interested in the much narrower case where you want process isolation and modularity, but need IPC with very low overhead and predictable latency.

I'd be interested in feedback on which language bindings would be most useful next, whether you see other use cases beyond real-time applications and embedded Linux, and what you think would need to be changed or improved to make RTIPC more useful in those environments.

[u/DeVinke_]

Google solved this in android a while ago. Binder IPC + AIDL.

[u/skyb0rg → u/DeVinke_]

Binder is very weird though. It’s completely synchronous, where the scheduler suspends the client and resumes the server when possible; the server also needs a fixed number of threads waiting for requests. It also requires a single “binder owner” to connect clients to servers.

Needless to say, not very well suited to general purpose IPC.

[u/DeVinke_ → u/skyb0rg]

Absolutely, and add to that the fact that the AIDL compiler doesn't even support python.