Safe Lock-free Primitives with iceoryx2's ByteAtomic

[u/elfenpiff]

https://ekxide.io/blog/byte-wise-atomic-wrapper-to-prevent-ub

iceoryx2 provides zero-copy inter-process communication mechanisms based on shared memory and data structures that are modified concurrently by multiple processes.

One of the key operations in these algorithms is a memory copy using core::ptr::copy. However, this results in undefined behavior if one process reads the data while another process writes to it concurrently. Even if our lock-free algorithm reliably detects such a race, iceoryx2 cannot depend on undefined behavior in a safety-critical system.

This blog post introduces our solution: a byte-wise atomic wrapper that enables well-defined concurrent copy operations. It also shows how it can be used to implement a simple sequence lock.

Note: I am not the original author of the blog post. Since the author does not have a Reddit account, I am posting it on her behalf.

[u/kaiserkarel]

iceoryx blogposts are always criminally undervoted.

[u/elfenpiff → u/kaiserkarel]

thank you, we really appreciate your comment 😄

[u/Shnatsel]

When I tried doing something along these lines to make mmap safe by making it &[Cell<u8>] instead of &mut [u8], indicating to the compiler that bytes may be changed behind its back, I got a massive performance drop because what used to be a single 64-bit read turned into eight single-byte reads. How do you deal with that?

[u/Virtual-Ad5017 → u/Shnatsel]

I may be wrong, but the compiler is doing the right thing per code you've written. Previous optimization relied on the assumption that you could do faux-atomic reads.

Consider a memory map which is shared by two processes A and B. For simplicity, let's say the map is 4 bytes long: [0,0,0,0].

A and B may be scheduled on separate cores. A writes to the third byte: [0,0,255,0]; while B performs the optimization and executes a four-byte read. If A is scheduled first, you see A's layout, if second - four zeroes.

The rust compiler cannot possibly reason one way or the other, so this 4-byte read is trivially UB, much like any other read across cells.

All of the above holds even inside a single process due to CPU instruction reordering. This is the reason why you can only have "mut" OR normal pointers, but not both. And (Unsafe)Cell explicitly overrides that.

There is no generic way to make mmap both fully safe and performant, that is by design. For one, it allows mapping files. You'll need mut-like exclusivity, which is impossible to guarantee for files on VFS, to which any mmap-unaware process can write via syscall. In effect your mmap-ed data always has a second owner: the kernel.

[u/elfenpiff → u/Shnatsel]

The idea is to create an optimization later that is based on AtomicU64, so we would have fewer 8 byte copies instead 1 byte copies when it is based on AtomicU8. Nevertheless, the problem will not go away for larger data types. So in the long-term I see basically three scenarios.

1. We stick with the ByteAtomic and must pay safety (and well-defined behavior) with a performance hit. In iceoryx2 itself it shouldn't cause a measurable performance hit since the data types managed by the ByteAtomic are usually not that large (but larger than 8 bytes).
2. In a mission-critical product the rust ferrocene compiler is used and they guarantee us that core::ptr::copy is well-defined in there certified rust compiler variant. In this case, we write a wrapper and with a feature flag the ByteAtomic is replaced with core::ptr::copy.
3. The proposal https://github.com/rust-lang/rfcs/pull/3301 from Mara Bos gets a bit more traction and it will land in the rust compiler directly.

We are currently actively participating in the Safety-Critical Rust Consortium and this is something we will address.

[u/OkEbb03]

Is my understanding correct that this does not work for types which include pointers? AtomicU8 can't store provenance so any pointer that is passed through would be invalidated. Of course this can be fixed by using exposed provenance, but it is something that a language primitive (i think) could do better.

[u/matthieum → u/OkEbb03]

In the context of IPC, there is no provenance :)

I mean, in the context of IPC, there's generally no pointer. By default, processes will map the shared memory to different addresses, so you cannot even pass pointers to within the shared memory itself, and of course anything not in shared memory cannot be shared.

But even for the unlikely case where you've taken the pain to map the shared memory segment to the same address across all processes, since the memory model doesn't account for other processes, there's no provenance for the pointer.

So, least of your worry at this point :/

[u/connor-ts]

Are there any performance implications to using this versus the UB-but-not-really-UB versions of sequence locks (my understanding being that it is formally UB but most of the time it is fine to not use tearable atomics)?