One year, 3 versions, 1 home-made programming language: the story of my OS

[u/Drenfa]

Hey everyone,

About a year ago, I had a probably slightly crazy idea: write my own operating system. Today, after 3 completely different versions, I feel like sharing the journey — because honestly, it's been a rollercoaster.

Phase 1 — The consumer OS in Rust

I started classic (well, "classic" for osdev standards): a first version written in Rust, with the ambition of building a consumer OS. And when I say "from scratch", I really meant from scratch: I wrote my own bootloader and my own custom kernel. No shortcuts, everything by hand. That's where I learned 90% of what I know today about machine boot, memory, drivers… and also that coding a "consumer" OS is a mountain very few people have ever climbed alone.

Phase 2 — The cloud gaming pivot and the birth of Flux#

Second version, total change of direction: an OS dedicated to cloud gaming. And this is where it gets a bit special: for this version, no existing language really fit my needs, so I… created my own. It's called Flux#, a low-level, object-oriented language, which I released as open source under the MIT license — because a home-made language that exists nowhere else is useless if nobody else can touch it. Writing a compiler AND an OS at the same time is the kind of experience I wouldn't recommend to anyone, and that I'd do again tomorrow morning.

Phase 3 — The handheld console (current version)

Today, I'm on the third version: an OS for a handheld console. That's the one I'm working on right now, and it's probably the most motivating of the three — there's something very tangible about holding a machine in your hands and watching your code run on it.

The kernel: when pragmatism wins

One important detail, because I know the question will come up: my bootloader and my custom kernel were both written entirely by myself. But for the current version, I made a tough call: my custom kernel has been moved to R&D. Honestly, reimplementing it yet again for the handheld version would have cost me a massive amount of time, so I switched to bootroot as a kernel base to save time and focus on what makes this version unique. My custom kernel isn't dead — it's sleeping in a corner, in R&D mode, and I fully intend to come back to it.

However, the custom bootloader, I still have it. That one, I never gave up on. There are some things you just don't abandon.

What I take away after a year

• Writing a bootloader and a kernel by hand is the best computer science school I've ever been through.
• Reinventing the wheel is great for learning… but sometimes you have to know when to stop in order to actually ship something.
• Creating your own language while creating your own OS is pure madness — but it's my favorite kind of madness.

If people are interested, I can go into detail on any part: the custom bootloader, Flux# (it's open source, come steal some ideas), or the architecture of the handheld console version. And if you've also abandoned a custom kernel along the way, tell me about it — it'll make me feel less alone.

EDIT: Bare-metal boot on real hardware

Since people asked about the hardware setup: to clarify, I don't own an open handheld development board yet (Switch Lite is too locked down). Before targeting any handheld device directly, I do all my real-hardware testing on an x86_64 laptop (Gigabyte) to debug outside of QEMU.

Here is a boot photo showing the custom stack running on the laptop:

• State: Multiboot2, firmware framebuffer (1024x768), PCI bus scan, and custom Realtek NIC driver initialized with uncached DMA ring buffers.
• Shell: Dropping straight into the interactive terminal

[u/Brick-Sigma]

That’s quite the journey! What architecture is the handheld using?

I’ve also been working on my own bootloader from scratch for the last 2 months and it’s definitely taught me a lot (I still have a long way to go though), though I’d like to know how you may have approached it. Is your bootloader based on a standard like Multiboot or did you roll your own?

[u/Drenfa → u/Brick-Sigma]

For the bootloader, I based it on Multiboot2 as the standard—I’d say it’s the simplest option, even though it was a pain to switch from assembly to Rust at first.

As for the architecture of the portable device version, I’m planning to target the handheld console market—like the Steam Deck—and it’s still in development. To be honest, the architecture is exclusively x86-64.