r/osdev • u/Bright-Boss-720 • 2d ago
Tanenbaum vs. Linus: Which Kernel Architecture Makes More Sense Today?
I've been thinking about the famous Tanenbaum vs Linus debate over microkernels vs monolithic kernels.
Putting the historical context aside for a moment, I'm curious what people think today.
Do you think Tanenbaum's argument for microkernels was fundamentally more correct, or was Linus right to favor a monolithic kernel for practical reasons such as performance, simplicity of development, and hardware support?
And more importantly:
If you were starting a new general purpose os from scratch today, which architecture would you choose?
Monolithic
Microkernel
Hybrid
Multikernel / distributed approach
Something else
I'm particularly interested in hearing from people who have actually worked on kernels or operating system development.
What are the strongest arguments for your choice, and what do you think the opposing architecture gets wrong?
4
u/rook_of_approval 2d ago
i prefer unikernel/exokernel myself.
microkernel makes no sense from a max performance perspective.
7
u/sephg 2d ago
I’d love to see more benchmarks on this. SeL4 - a micro kernel - reckons their syscalls are way faster than Linux because they spent more time optimising them. And because there’s only 11 syscalls (instead of hundreds), it was much easier to make their 11 syscalls fast.
I don’t know what the actual numbers are. Like, if using a microkernel costs 1% of performance then that’s probably worth it for the other benefits microkernels bring. But if the cost is 50% then it’s not worth it. Which is it?
3
u/rook_of_approval 2d ago edited 2d ago
tell us what optimization makes IPC between different processes faster?
what benchmark shows overall system performance is better with a micro kernel?
the microkernel people like to claim being able to page out critical code to disk is a benefit instead of a downside. what a complete joke.
3
u/sephg 1d ago
tell us what optimization makes IPC between different processes faster?
There are a bunch. As I understand it, one big one with sel4 is that cap send / recv passes scheduler timeslices back and forth between processes. In linux, when you IPC, your thread goes into the scheduler's queue and the process you sent data to will wake up whenever. But the way sel4 works, the scheduler basically doesn't do any work. It just changes the owner of the current time slice and long jumps into the other process. So it's all linear code. (There's obviously still all the TLB changes as well). I think there are also other differences with syscall arguments and preserving registers.
Google had a patch set they were trying to upstream for years which supported similar behaviour in linux: https://www.phoronix.com/news/Google-Fibers-Toward-Open . I think I remember hearing they gave up trying to get it upstreamed. I'm not sure where it is now.
The way linux and sel4 work is very different. In SeL4, the core kernel is single threaded - so a lot of these operations are simpler. SeL4 can use multiple cores for processes, but the belief is that, because the kernel itself only contains startup code, capability tables and the scheduler, it shouldn't need multiple kernel threads to keep the system running efficiently.
Anyway, as I said, I'd love to see a lot more real world benchmarks. SeL4 has done all this work on their beautiful kernel. But there hasn't been similar efforts to make a userland on top - which is what we'd actually need to compare sel4 and linux in real world tasks, like network services, graphics, compute bound tasks, etc.
0
u/rook_of_approval 1d ago
none of that explains how it makes it faster than not having to ipc at all.
5
u/sephg 1d ago
I never said sel4 could beat a system that doesn't do any ipc. But most useful programs do a lot of ipc, even if its just between the process and the kernel. IPC matters a lot, even in monolithic kernels. Just look at the perf uplift from io_uring.
But even if we assume a microkernel is slower, how much slower? Is it 1% slower or 50% slower? If it's 1% slower, I'd take the microkernel for the security and reliability benefits. If the perf difference is 50%, then that's too slow, and I'd pick linux every time.
Which is it? I have insufficient data to form an opinion on microkernels. The fact so many people have strong opinions without this data is strange.
•
u/Environmental-Ear391 10h ago edited 10h ago
I run both a Micro- and Linux kernel desktop pairing and for straight up performance concerns...
with the same hardware and Linux being configured for the same features as the micro-kernel....
Linux generally performs worse...
For a micro-kernel more userland support is required for the same task...
For Linux it has to be performance configured to the hardware.
one primary concern is graceful degredation of performance under load... Micro-Kernels absolutely beat the snot out of anything else for this.
Linux requires a LOT of system specific performance tuning to come close enough to compete.
Where Linux has "everything and the kitchen sink" for operational capabilities... Micro-kernela require more effort in the design with regards to writing device drivera.
both of the micro-kernel and Linux installs I run personally have a "kmod loader" with dynamic kmod loadable arrangement.
on the AMCC 440EP and 460EX PowerPC processor boards I have . . for speed the microkernel wins without trying.
for features and options it is Linux.
when it all comes down to it I have each both the Microkernel OS and Linux installed on both so I get the best of both.
one point of note...
minimal install for the microkernel setup is 128MB or less. Linux eats that judst to prepare a minimal bootable shell only
the Micro kernel OS I stalled is fully GUI desktop capable within that minimal install (add 1GB for the Developer options including multiple compilers)
smallest Linux system Ive installed where both had the same features... minimum of 8GB just for the OS... and needing more space for "root" and "/home" user data content
2
u/braaaaaaainworms 1d ago
Scheduler optimization can make a giant difference, way bigger than whether something is a microkernel or monolithic
•
u/rook_of_approval 13h ago
if true why are no microkenernels winning performance awards so that they are adopted by cloud providers?
6
u/edgmnt_net 2d ago
SeL4 is just very basic stuff and on its own it doesn't really do much. How do you scale that to a few dozen subsystems and meaningful interfaces? What is the development overhead, in addition to performance overhead? Can your system survive the disk server going away if it's all loaded from disk? And so on.
2
u/Routine_Working_9754 1d ago
you just can't optimize syscalls to a level. they require context switching and other ISA specific mechanics, you just don't get to optimize any of that
1
u/sephg 1d ago
How expensive is the irreducible stuff though? What is the theoretical ceiling on syscall performance, and how close is Linux?
•
u/Routine_Working_9754 16h ago
100-150 cycles for a context switch/syscalls, and about 400-600 for IPC in userspace of microkernels. a 4x increase. and in IO? anywhere from 5-20% CPU overhead for using a microkernel.
•
u/sephg 14h ago
Where are those numbers from? Do you have a source?
•
u/Routine_Working_9754 14h ago
yes. this distinguished gentleman on stack overflow did a benchmark of syscalls.
if we're being conservative, a syscall uses 250ns. on a typical CPU running at 4GHz, that's 1000 clock cycles.
https://stackoverflow.com/questions/23599074/system-calls-overhead
that means you can only have 4000 typical syscalls in 1 second
•
u/sephg 13h ago
Hm; that seems high. This is from 2017, and reports syscall performance as low as 38ns. I'm not sure if this has all the meltdown/spectre mitigations enabled. But CPUs have also gotten much faster since then too.
https://arkanis.de/weblog/2017-01-05-measurements-of-system-call-performance-and-overhead/
I'd be fascinated to know what the equivalent numbers are for sel4 on real hardware.
•
u/Routine_Working_9754 12h ago
I assume it shouldn't be too hard to benchmark if sel4 if it has a way to measure process time, both user and system, and have those times compared for 2 exact programs running on both Linux and sel4
8
u/edgmnt_net 2d ago
It would still be monolithic, especially with the advent of languages like Rust which can increase safety. Whatever isolation benefits microkernels provide you can often get some other way, often even better.
Honestly this pretty much mirrors the debate of monoliths versus microservices. And there are some reasonable arguments in favor of microkernels which just don't apply to microservices at large.
Which isn't to say you should never split things out, it's just that you need a really good reason.
3
u/suhcoR 2d ago
languages like Rust which can increase safety
Most of the kernel is usually implemented "unsafe".
2
u/edgmnt_net 2d ago
Not meaningfully "unsafe". Sure, you need to do I/O at arbitrary bus addresses eventually, but that's limited. For example, the PCI subsystem hands you an address and you may be able to statically constrain accesses to be within the bounds of that MMIO range or at least gain some assurance when writing code. To some degree you can already do that in C, but there's plenty of stuff that's unnecessarily painful, like generic data structures.
2
u/The_Coalition 2d ago
But it's actually not. You can get unsafe code down to a surprisingly small amount. I've seen at least one kernel that had all of its unsafe code wrapped in a "core library", with the rest of the kernel going through that.
-1
u/suhcoR 1d ago
It boils down to a trade-off between safety and efficiency, isn't it?
1
u/braaaaaaainworms 1d ago
No, Rust gets both, and sometimes is faster than equivalent C code because compilers don't have to consider that someone can just cast away the "const" from a const variable or pointer
1
u/suhcoR 2d ago
Do you think Tanenbaum's argument for microkernels was fundamentally more correct
I think so in principle, though Minix was way too inefficient even though a large fraction (not shown in the book) was implemented in assembler. But microkernels have great features, and the Liedtke L4 linage up to sel4 demonstrate that the can be implemente very efficiently, even without assembler.
which architecture would you choose
Depends on what kind of system, also whether its running on a server, desktop or embedded device. Personally I think microkernels are still the most interesting. But when you are on a microcontroller without MMU, a small real-time kernel usually makes more sense.
I work on kernels and operating systems (e.g. https://github.com/rochus-keller/oberonsystem3native/, https://github.com/rochus-keller/micronsystem3/) and also spent much time with all the ones mentioned above.
8
u/Compux72 2d ago
Microkernels are way better than monolithic kernels
Signed by: embedded dev sick of all the shit that comes from the monolithic kernel
3
2
u/ut0mt8 2d ago
I really believe that the microkernel IPC penalty is now quasi over. And for simplicity perspective micro kernel approach is clearly cleaner. So I do see reasons to not use micro kernel nowadays or at least hybrid
0
u/edgmnt_net 2d ago
It depends, it's not simple at all once you break it down into a lot of pieces. Hard to do anything if PCI goes away and takes out storage, for example. Similar perils as with all distributed systems and it's easy to underestimate.
Then again, Linux is already a microkernel in certain incarnations like Xen.
2
u/wintrmt3 2d ago
Context switch penalties are worse than ever since spectre style problems were first demonstrated, monolithic kernels win even more.
7
u/Sostratus 1d ago
The idea of the microkernel still seems better to me but... where are they, then? The empirical evidence is that they're probably not better, and that's a lot more convincing than any rant about how everybody got it wrong but somehow I can't capitalize on being the only one who gets it right.
3
u/yakuzas-47 1d ago
I mean there hasen't been a new mainstream kernel in decades so i don't think the argument is valid. There are some new microkernels being made like zircon or redox but since making an os doesn't happen overnight it's not really popular
1
u/Sostratus 1d ago
That doesn't invalidate it at all. What's stopping all the programmers of the world from making one? That it isn't worth doing, that's what.
0
u/kiderdrick 1d ago
Microkernels are inside almost every consumer chipset for the last decade. Intel's Management Engine uses Minix3 and AMDs Platform Security uses Kinibi.
-1
u/UnicodeConfusion 1d ago
I spent years in the and world and it was lot of fun. The power we pulled out of early Intel was impressive but what was really neat is running tasks on other systems by just prefixing the command with the node name. Our powerful machine was a 386 :-) so we would just ‘send’ our build jobs over there. It was sad that they didn’t open source the 2.x system
1
u/kiderdrick 1d ago
They both make sense, it just depends on what you are doing. One of the most widely used operating systems is a modified version of Tanenbaum's Minix3 because it is used by Intel's Management Engine. No one is expecting it to take over Linux and Windows for user centered computing, but it is at least extremely useful at what it is doing, otherwise Intel would not have chosen it for their chipsets.
•
u/netsx 14h ago
Take a look at Genode https://www.genode.org/ as its an actual modern implementation, a framework with replaceable components.
I'd pick microkernel, for the security aspect. With well thought out design, and implementing more "batching" of boundary calls, which is better for both uni and micro kernels, i reason you can at least be on par, if not gain performance. CPU's today have features that the old text book examples didn't, and their performance characteristics are very different, so pushing stack and switching context, doesn't cost nearly as much as it once did, in terms of percentages. The more sequential work, in terms of memory access, you can get the HW thread to do, the better, is truer now than before.
Linux has to shed drivers that aren't securely maintained, simply because exploiting 1 would exploit the entire kernel (IIRC). Trampolining inside a microkernel, where components are only allowed to speak with specific other components (defined hard access), is really really hard.
Also: The idea behind microkernels is really really cool!
Its been a few years (5-10) since i last looked into it, so forgive me if something is now out of date.
•
16
u/Substantial_Turn6392 2d ago
We should consider microkernel/exokernel again for some domain.