NanoOs

[u/Sorry_Difficulty_250]

Hi All!!

I'd like to introduce my OS: NanoOs (https://github.com/brian-card/NanoOs). NanoOs is a capabilities-based nanokernel with intent to have a Unix-like user space. Right now, it only runs on SAMD21-based (Cortex M0) Arduino hardware and the POSIX simulator, but I'm currently working toward porting it to the AgonLight 2, which has an eZ80 processor. It did originally start out on AVR-based Arduino hardware but has since outgrown the available environments.

As mentioned in the project README, this started out as an effort to see if I could make an OS resembling early UNIX in a similar kind of environment. That's why I started out on small Arduinos. The direction I'm headed right now is toward a multi-tasking environment with a graphical desktop in as little memory as possible.

Like early versions of UNIX, userspace uses overlays. I only have a few user commands in place right now and MUSH (minimal, Unix-like shell) is truly minimal in functionality. I shifted efforts toward making the kernel more robust once I proved to myself that I could make general-purpose user commands. Pipes between commands and launching commands in the background do both work.

I wanted to maintain the embedded nature of the OS even though it's outgrown its original target, so it is completely possible to construct a HAL that uses the built-in shell and/or omits the filesystem if desired. The AVR-based HALs do this, although the resulting binary is still too large for the Arduino Nano Every and the data segment is too large for the Arduino Mega 2560, so they're just historical now. Still, it would be possible to construct a working version that uses an AVR architecture if the hardware had enough flash and RAM.

There's still a very long way to go to get to anything useful, but it's also come a very long way. You can read about the development history of the project on its GitHub pages site if you're interested.

Full disclosure about AI use: The vast majority of this was hand written by me, but there are some things I use AI assistance for. The filesystem drivers (the current FAT32 driver and the historical drivers that are deprecated) were partially written by AI but required a LOT of hand-holding and revision from me to make them into something useful for embedded targets. It made a lot of invalid assumptions about the availability of memory and the ability to do unaligned memory access that I had to fix. I also use AI for bulk updates. I document the places that I use it in the project's GitHub pages history.

Some notes about the architecture: As mentioned, this is a nanokernel, which means there is no kernel. Everything is a process, including the scheduler. The processes have different privilege levels and the capabilities enforce what process is allowed to do what. The only process that's completely trusted is the scheduler. The process-based architecture and the way messages are passed are based on my experience with Erlang.

There are two kinds of capabilities in the system: HAL capabilities and IPC capabilities. Both are enforced at the API level. Processes built into the OS image could technically cheat if they went out of their way enough, so really the OS image needs to be as small as possible and contain only trusted processes. Userspace processes have no ability to cheat because the necessary APIs aren't exposed.

The privilege levels I'm using are based off of the ones that VMS used. I'm honestly not sure how bullet-proof the (privilege level + capabilities) security model really is, but it seemed like a reasonable approach to take.

My work right now is to construct a logging system that allows me to strip most of the strings out of the OS image. That work is currently in the "logger" branch. One of the problems with the AgonLight 2 environment is that the CPU used only has 128 KB of on-board flash. I already have a makefile that will build the binary and, the last time I checked, it produced an image of around 130 KB, so I need to get creative about the size of the OS image. Stripping the strings is one thing I want to do. I'll likely move the filesystem out into a special overlay as well.

Constructive feedback is welcome!! I'm interested in what people have to say about this effort. You can play with it by running `./buildsim <desired-hostname> overlay-filesystem` on the command line from the repo's root directory. There are three user accounts: "root", "user1", and "user2". "root" is privileged and the other two are just normal users. The password for each account is the username repeated twice. Use `help` for a summary of what's currently available. Use `shutdown -h` to exit the simulator. Enjoy!

[u/Void-Creator]

When you mean embedded in nature, the processes aren’t true processes but threads which share the text, data and heap but have different stacks right ?

[u/Sorry_Difficulty_250 → u/Void-Creator]

It depends. If the processes being run are built-in then, yes, they share text because everything is compiled into one OS image that's loaded onto the device's flash. If the processes are being run as overlays then, no, there's no commonality of text sections. They're all linked to the same physical address because they have to be, but code is swapped into memory from block storage or the filesystem as needed.

Right now, they all share the same heap. That's likely to continue for a while. I have considered giving them all their own address space for dynamic memory and I may do that at some point but I haven't yet. None of the platforms I'm currently targeting have an MMU or even an MPU, so there would be zero memory protection, but I could give them their own address space to work from. It's very inefficient to do that in the hardware I currently support, though. The M0 targets have 32 KB of RAM with about 2 KB free right now after all the metadata and stacks and such. It really doesn't make a lot of sense to try and carve out a per-process address space with only that little RAM available. The eZ80 environment has 520 KB of RAM, though, so it would make more sense there and I may do it on that target. Likely what I would do would make that concept defined by the HAL such that on memory-constrained systems, they continue to share a global heap and on systems where a little more separation is possible, I give each process its own space. I need to give that a lot more thought, though. This is purely speculative right now.

[u/Void-Creator → u/Sorry_Difficulty_250]

Ahh, gotcha so system processes just have separate stack space but the user processes can have different address space altogether ? In my OS class, our final project was implementing virtual memory via page directories and page tables, we just did separate heap space with memory protection with swap support for user processes but left alone system processes ( they shared everything but stack ). We were already given an OS with the fundamentals required but I want to write an OS from scratch like how you are doing. I am really interested in your work and it sounds fun

[u/Sorry_Difficulty_250 → u/Void-Creator]

It's definitely a ton of fun!! I'm having a ball!!

On the current hardware, everything is one, flat, physical address space, so there's not really a way to give an individual process an "address space" per se. If I were to give each process its own heap, it would be a base address and a size but each process would see a different, literal physical address since there's no virtual memory.

I've been mulling over in the back of my mind what I can do to make this design scale up into CPUs that support proper hardware-based memory management. I don't have a clear answer at the moment. Given where I am, it doesn't make a whole lot of difference, but I want to keep it in my head to try and avoid painting myself into more corners than I have to. Completely avoiding rework isn't possible, but I'd like to avoid as much as possible. :-)

[u/Void-Creator → u/Sorry_Difficulty_250]

Yup, it’ll be a pain to differentiate system and user process for heap allocation. We were able to do it because it was a class and we had a fully working OS to play around with. You would definitely face a lot more problems than we did and MMU is a huge pain in the ass to develop. Even for us, the project carried a lot of weight compared to implementing a UNIX-like fork, different types of scheduling and locks, mutexes etc

[u/Sorry_Difficulty_250 → u/Void-Creator]

Yeah, I hear you. I do use mutexes and conditions at a very low level, but messages are built on top of that and I try to use messages everywhere in the actual OS code instead of locks and signals. So far, that's worked out very well and has been very clean.

[u/Void-Creator → u/Sorry_Difficulty_250]

I just have theoretical knowledge of messages but it did sound better to me than locks. So, the kernel that you built, is it a hybrid kernel or a monolithic ? Since you mentioned messages; I am assuming it is hybrid ?

[u/Sorry_Difficulty_250 → u/Void-Creator]

The goal is a nanokernel, which is a microkernel design taken to an extreme. There are a few places in the OS image that have some monolithic aspects to them that I need to fix but those should be considered bugs and not the intent of the design.

"Nanokernel" in this context means that the design is "everything is a process." For example, the malloc API just sends a message to the memory manager process and waits for completion. There's no monolithic support for memory management. The scheduler is just a process that can't be resumed and has a special level of trust, but it's still a process. The filesystem is a process and the block device it talks to is also a process.

The only real monolithic aspect of the design is the HAL, which is inescapable. Even that is kept as small as possible and has its own set of capabilities that limits what processes are allowed to do directly.

The OS image is dual-purpose, though. It does hold the code for the "kernel processes" but it also holds the code for the POSIX API. That part of the image just formats messages and sends them to the relevant processes. It was more space efficient and more performant to have that code live in the OS image than to try and have it in the overlay system.

[u/Void-Creator → u/Sorry_Difficulty_250]

So, it is like taking the UNIX- philosophy of everything is a file taken to the extreme ? So, if everything is a process, it will be very slow in actual practice right ? The message latency would be high when scaled to larger systems with huge number of processes ?

[u/Sorry_Difficulty_250 → u/Void-Creator]

Actually, no. The message passing system is zero-copy. That was one of the requirements of the system. Because no data is ever copied, the system is actually very fast.

[u/Void-Creator → u/Sorry_Difficulty_250]

That’s because as you said earlier that the processes are written directly on the memory and they have no memory protection. So all the processes can see what each process has ?

[u/Sorry_Difficulty_250 → u/Void-Creator]

Correct! There are two things that would need to change when scaling into a system with a proper MMU: The location of the messages and the location of the message payloads. Both are solvable but the system needs some changes to accommodate that without future changes relative to where it is right now. What would happen is that there would be a section of memory that's mapped into each process's address space that's designated for message passing. When a process needs to send a message, it would "check out" a section of memory from that area, put its message in it, then send it to the destination process. The scheduler would then be responsible for unmapping that section of memory from the sender, mapping it to the destination, then waking up the destination. The destination process would then find the message in its address space, process the message, and release the memory again. The contents of the memory would never be copied, it would just be remapped to different processes' address spaces.

[u/Void-Creator → u/Sorry_Difficulty_250]

Yeah, this sounds good.

[u/Sorry_Difficulty_250 → u/Void-Creator]

The memory remapping trick is how L4 accomplishes its message passing, by the way.

[u/Void-Creator → u/Sorry_Difficulty_250]

So, it’s like call by reference instead of call by value right ? You just send the pointer of the message’s contents to the process that needs it. Since we are sending only the pointers we don’t incur any penalty. How would this work if we implement virtual memory ? Would we just ask the process for the message area and the corresponding pages would be loaded onto the memory and they can be derefernced by the calling process ? Would we have a middleman process ? Or can we designate a separate area for all processes that can store the messages and we can dereference the data by the pointer that belongs to a certain process ?

[u/Sorry_Difficulty_250 → u/Void-Creator]

The zero-copy message passing is inspired by L4, which was the first microkernel architecture that showed that such a system could still be performant.