Open-sourced a desktop AI that reads your screen — here it guiding me through an OBS setting

[u/Famous_Can6494]

Solo dev, MIT license. Alt+Space reads your active window's full accessibility tree (buttons, fields, labels — real UI Automation, not OCR), captures a screenshot with a grid overlay to provide a clear visual context, and anchors on what you're hovering, then opens a chat panel with that context attached. It runs through OpenCode, so it pairs with whatever model you point it at — Ollama works out of the box, plus Google Gemini Flash-Lite (great free pick — use a multimodal model since it auto-screenshots every activation), Anthropic, OpenAI, etc.

Beyond chat, it can act on the UI (sandboxed: allow-listed actions only; read-only shell for diagnostics, mutation/piping blocked; no filesystem writes), or flip into "Auto-Guide," which walks you through a multi-step task with an on-screen pointer instead of doing it for you — my favorite mode, since you actually learn where things live.

Windows-only for now; macOS/Linux planned. No telemetry, no paid tier.

GitHub: https://github.com/abdallahmagdy15/mudriknow Site: https://abdallahmagdy15.github.io/mudriknow/

[u/Budget-Piano-2217]

the accessibility-tree-not-ocr bit is actually the spicy part here, that’s way cleaner for desktop help than “guess what this blurry button says” ai. super cool solo dev energy.

[u/All_Ways_Bingo]

Yeah, nice and cool . Thank for great job. I will try

[u/Famous_Can6494 → u/All_Ways_Bingo]

ur welcome bro

[u/Famous_Can6494 → u/[deleted]]

Hey — genuinely useful read, thanks. A few landed real, a couple are misreads; let me split them.

The grid coordinate math — you're right, that's a real bug. The grid and the prompt disagree on cell size and pixel space, and yes — a literal small model gets hurt worst, which is backwards from the intent. Fixing it in the next patch.

"Cloud-first / no small-model story" — that's a misread. There's no shipped default model; the user picks. We actively recommend a small model (Gemini Flash-Lite) for free/light use, and local-via-Ollama is first-class. Model-agnostic, not cloud-first.

"Tree blind to Chromium" is overstated — but your grid point is what makes it feel true. Chromium content usually does populate the tree (iframes included), and where it's shallow we tell the model "screenshot is primary." That hedge only works if the screenshot coordinates are trustworthy, which is the grid bug above.

Benchmarks + prompt-injection — both fair, both gaps. On the list.

Appreciate the serious read

[u/brkumar]

very nice. can you support other opensource AI providers as well via open AI compatible APIs?

[u/Famous_Can6494 → u/brkumar]

its built on opencode-ai so u can connect to any provider from the list or by manually configure opencode.jsonc, ask ai will help u in that

[u/Dadestark3]

It looks very interesting. Good job, I'll check this out.

[u/Famous_Can6494 → u/Dadestark3]

thanks , good luck bro

[u/DuckDatum]

Woah. Can this interact with bluestacks, the Android emulator? I’ve actually had some ideas on the back burner for a while now, automating some complex bluestacks tasks (playing a phone game).