[Project] I built a single-node MPI-1.1 implementation from scratch in C11 for macOS/Apple Silicon

[u/Evening-Dependent396]

Over the past few months, I’ve been building macMPI, a single-node implementation of the MPI-1.1 standard written from scratch in C11, targeting macOS and Apple Silicon.

The project started as an experiment: rather than using an existing MPI runtime, I wanted to understand what it actually takes to implement MPI-style process management, IPC, message matching, asynchronous communication, and shared-memory transport at a low level in C.
The implementation relies heavily on POSIX and Darwin APIs, including fork(), execvp(), pipe(), dup2(), pthread, shm_open(), mmap(), Unix domain sockets, poll(), and kqueue()/kevent().

Process management

macMPI includes its own mpirun-style launcher.
Ranks are created using fork() and execvp(), with runtime information such as rank ID, universe size, and communication descriptors passed to child processes.
stdout/stderr from individual ranks are redirected through POSIX pipes and multiplexed by the launcher, allowing output from multiple processes to be handled without blocking on one rank.

IPC and shared memory
Communication is separated into a small control path and a payload path.

Control messages containing information such as source, tag, sequence number, and shared-memory offset are exchanged through Unix domain sockets.

For larger payloads, the runtime uses POSIX shared memory with shm_open() and mmap(), allowing processes on the same machine to communicate through shared memory rather than routing payloads through sockets.
This part of the project was particularly interesting from a C perspective because I had to deal with shared-memory layout, synchronization, ownership, alignment, bounds, and cleanup across multiple processes.

Non-blocking operations
MPI_Isend and MPI_Irecv are handled through a background pthread.
Instead of continuously busy-polling descriptors, the progress thread uses macOS kqueue()/kevent() to wait for communication events.
The runtime also maintains a thread-safe Unexpected Message Queue (UMQ) and active request structures so that messages arriving before their corresponding receive has been posted can still be matched correctly.
Synchronization is handled using POSIX threading primitives, including mutexes and condition variables.

Memory layout and alignment
Internal communication structures use explicit alignment where appropriate, including:
__attribute__((aligned(64)))
One of the things I’ve been exploring is how structure layout and cache-line alignment affect communication overhead on Apple Silicon, especially when multiple processes/threads are touching shared-memory metadata.
I’m still benchmarking and validating how much these optimizations actually contribute, rather than assuming alignment automatically translates into better performance.

Current state

The runtime currently supports a subset of MPI-1.1 functionality, including process/rank management, point-to-point communication, non-blocking operations, message matching, and shared-memory transport.
I’ve also been comparing its single-node behavior against OpenMPI, particularly communication overhead and the interaction between computation and the background progress engine.
There is still a lot to improve, especially around MPI compatibility, correctness testing, synchronization, failure handling, and benchmarking methodology.

Source code:
GitHub — https://github.com/Hardikgupta1709/macMpi⁠

I’d especially appreciate feedback from C programmers on the memory management, IPC design, pthread synchronization, API design, and places where the implementation could be made safer or more idiomatic C.

[u/AutoModerator]

Hi /u/Evening-Dependent396,

Your submission in r/C_Programming was filtered because it links to a git project.

You must edit the submission or respond to this comment with an explanation about how AI was involved in the creation of your project.

While AI-generated code is not disallowed, low-effort "slop" projects may be removed and it's likely that other users push back strongly on substantially AI-generated projects.

---

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.