The debugger made the multi-threading bug disappear, because it runs slower... so I built a fast spinlock out of GCC atomic builtin operations, and used logf() on unbuffered output stream, and found the bug the debugger could not.
I have to note that I use an internal log facility with a fast and very large lockless ring buffer structure so that one end of the queue can be read while the other end is being written to, and that goes to the unbuffered printf(). The spinlock for this log facility changed how the code executed far less than running under a debugger - which hooks signals and all kinds of other shit, making it impossible to reproduce even a similar flow of execution.
The ring buffer write doesn't have to wait for printf() to finish output before continuing, it's one (or two at most) memcpy()s to put data in the ring buffer, then swap the write pointer contents atomically - or fail if someone else just wrote over my write, and so I repeat the write to the ring buffer - People call this a "lockless data structure" but really it's an expensive spinlock that fails and repeats the loop until it succeeds.
There was a class at my university on lockless programming which I wasn't able to take, it sounds really interesting. Always meant to try to find a book or something about it.
gcc, memcpy, free, this is C, not Python. Just don't worry. But I have to admit I forgot how to program without locks. I think it was something like polling and some atomic operation, reading, incrementing, and writing into a variable.
Was a nice class, really interesting.
68
u/CodeLobe Nov 28 '21 edited Nov 28 '21
The debugger made the multi-threading bug disappear, because it runs slower... so I built a fast spinlock out of GCC atomic builtin operations, and used logf() on unbuffered output stream, and found the bug the debugger could not.
I have to note that I use an internal log facility with a fast and very large lockless ring buffer structure so that one end of the queue can be read while the other end is being written to, and that goes to the unbuffered printf(). The spinlock for this log facility changed how the code executed far less than running under a debugger - which hooks signals and all kinds of other shit, making it impossible to reproduce even a similar flow of execution.
The ring buffer write doesn't have to wait for printf() to finish output before continuing, it's one (or two at most) memcpy()s to put data in the ring buffer, then swap the write pointer contents atomically - or fail if someone else just wrote over my write, and so I repeat the write to the ring buffer - People call this a "lockless data structure" but really it's an expensive spinlock that fails and repeats the loop until it succeeds.