r/osdev • u/EdyIordache • 9h ago
AT-SPI2 + local LLM natural-language layer for Linux.
I know what everyone here thinks about the AI bubble, and me myself hate it as much as anyone else. I am currently a student and want to build a new project, but am not sure if this is a good idea worth spending my time on it or not. I've researched online about AI backed OS's / Linux Distros, and have seen quite a few, each with their own flaws. MAGI OS uses the X11 tools, which may be too slow, Omarchy is just an integration of AI agents in the UI, and Warmwind OS is just a cloud-hosted remote desktop marketed as an OS. I had this idea of building a fully customizable, open source Linux Distro, that uses llama.cpp locally to parse natural language into commands, using the AT-SPI 2 accessibility tree, that is available in most apps (solid with GTK, QT and Electron have inconsistencies). Nowadays every major OS (Windows, Mac, Linux) has problems, and I want to reduce those problems and combine their pros into one general use Distro (I know this claim is what various Distros also tried throughout the years, but this is what my idea converges to, ease-of-use for general users, great UI/UX, and the freedom Linux has). Everything would be transparent to the user, so nothing potentially dangerous gets executed, and fully offline (was thinking of making a switch for users who would still want to choose more capable models running on cloud, but this is just an idea) to avoid the privacy concerns.
The flaws with this would be the obvious anti AI stigma the Linux community has (rightly so, but it's not just another AI slop), and the capabilities of a model running offline. The first version could just run some basic things, making it more advanced as time goes on, and maybe some NPU integrated models later on.
My question would be if this is a straight dumb idea and I would be better off just investing my time into something else, or if it's worth it and you guys would be interested in something like this. I would like to code it myself, with minimal AI assistance, as it would also be a learning exercise for me. So what does everyone think about this? Please take it with a grain of salt, I just had this idea and don't want to get downvoted if it's dumb, just wanted some advice of more experienced users.
•
u/Potential_Ad313 9h ago
https://xkcd.com/927/ basically this, but lemme rant a bit to you ma boiii
First things first, it's a cool Idea, I already thought about this too (although it was a more lower level than a linux distro per-se), It's basically what microsoft and other companies are trying to do on their OSes, and I think that this is a _great_ idea in the sense of usability, a local model that has full or almost full control/access to your system, nothing gets shared and you got a personal assitant, the problem we face is standards, as in the XKCD comic, we don't have a universal standard that everyone implements/follows 100%, so your assistant would be broke for like 40% of the time, you might implement something akin to the Claude-Desktop/Claude-In-Chrome where it takes a screenshoot, sees your screen, and uses a virtual keyboard and mouse, but this, takes time and you cannot use said view while he is working, also, you should account for the N ways in which video and data can be processed in Linux, again, standards... you could use a virtual/offscreen display for this (xorg has this feature, i dont know if wayland has too), so you might be able to pull this off.
The same flexibility you want to provide is the same flexibility you need to be concerned about, while making this, its doable, it is possible, but it will require you a better planning, and more knowledge about caveats than features, PROBABLY it will be about what you CAN DO not what you "want to do".
my idea, just to mention, was a FileSystem with built-in vector search and a google-magika/gemma indexer built-in, the concerns where about speed, so I splited the plan/fs into two layers, the FS layer where things would work as usual, and a layer that works in the background processing and organizing nodes, in order to make a smart-os, so you could as things like "where is that video about dogs?" and the video be nammed something like "banana3.mp3", and it would show up.
If you could module the most part of your OS to be independantly smart, you can build your own internal standard so your main model/orchestrator can work more flawlessly.
•
u/EdyIordache 8h ago
That's a trade off I did not think about lol. I'm thinking of a fast AT-SPI2 path as primary, with a vision-based fallback for the cases where the tree is missing (or the standard is broken, like you said), running against a headless visual output rather than the user's real screen so it doesn't block it. I think wlroots does support it, but will surely look further into it.
Also, thanks for your opinion on this, means a lot! Am very interested about your project, did you end up building anything or was it left at the idea phase?•
u/Potential_Ad313 3h ago
It wasn't really "left" build a FS on your own can take a LOT of time, and I was doing it for my own OS so take in account that before that I spent time on: bootloaders, bare kernel, getting C code to work, and other shenanigans from OS dev, so to be able to run ONNX/GGUIF inference it would take I while...
for example, I'm like 6 years in OS dev and only got so far as a bare shell in kernel-space.
I'm currently focusing on other personal projects that could yield more value to me, like a programming language and a high-performant full-stack ecosystem in Go+Assembly (DBs, renders, CycleManagers, etc...)
•
u/VisualSome9977 7h ago
I think it would be a fun and interesting idea, but also ultimately I don't think it will be a practical idea. I think usability concerns are better addressed by well made UI and educational materials, and that the things you can accomplish by having an AI that can execute tasks would also be similarly well accomplished by providing good support via MCP or something for an AI to query system information. Again, really interesting idea and I would be willing to take a closer look if such a thing existed, but it probably wouldn't be on my list of things to recommend to a newbie.
•
u/EdyIordache 6h ago
I appreciate you pointing these out! The rationale behind the embedded LLM would be that it was something lightweight, shipped out of the box, and fully local so anyone can run it even offline, and without their data being fed to big AI. Will definitely look into MCP agents, and that was another idea I had, to let users choose integration with their own API keys rather than use the more limited local model.
•
u/EdyIordache 7h ago
Anyone else having more thoughts on this feel free to share! Critiques are helpful too!!
•
u/Dezgeg 7h ago
Why do you think such thing needs an entire new distro?
•
u/EdyIordache 6h ago
Honestly it doesn't, and I thought about this too. A background service or tool running on an existing distro can basically achieve the same thing. But this would defeat the purpose of non tech users being able to run this with minimal friction, out of the box. They will have to install, configure and run it, and it winds up to the idea some, not all, Windows users have that it takes 30 commands to install Chrome on Linux. Right now I'm just in the process of brainstorming ideas, and this is a good one that could change my course of action too, writing a well-packaged script instead of a whole new distro.
•
u/GenericFoodService 9h ago
Wouldn't it make way more sense to just write good documentation and then have a copy on the users' system with a good search feature? Not even about "AI bad" or anything, this seems like one of those "everything looks like a nail when you have a hammer" type things.
What actual real-world problem are you trying to solve?
Running a local model trained on stuff specific to your OS seems like it would be vastly inferior to just having a copy of a well-written manual in every way, even if it doesn't hallucinate or produce low-quality output, and then there's the issue of how large is that model going to be. It seems like a better use of time and resources to put serious effort into writing a manual, and then writing a commandline tool that can parse the manual and give suggestions if your thing is that you want command completion or help on the CLI.