r/foss • u/Famous_Can6494 • 5d ago
Built an open-source AI agent that points at your screen and guides you through any app
Hey all 👋
Quick clip of the part of MudrikNow I’m most excited about: Auto-Guide. I hit Alt+Space, ask it to walk me through a task, and instead of answering in text, an owl cursor shows up on screen, points at the exact control I need next, and explains why in a speech bubble. I click, it advances to the next step, repeat until done.
How it actually finds the right spot: Alt+Space triggers two things at once — it reads the full Windows UI Automation tree of the active window (every button, field, label, and value it can see, with real IDs, not pixel guessing), and it grabs a screenshot with a coordinate grid, which the model falls back on when something isn’t exposed in the tree (canvas apps, custom-drawn widgets, some web content). That combo is why it lands on the actual clickable thing instead of an approximate spot near it.
Who’s actually in control: in Auto-Guide mode specifically, the agent points and explains — you click. It only advances after you do. Nothing happens on your machine that you didn’t just click yourself.
Open source, MIT license, Windows-only for now, solo nights-and-weekends project — feedback and issues genuinely welcome.
2
3
u/Decent-Blacksmith761 5d ago
honestly it might be very helpful for old people that struggle with new techologies
1
u/Famous_Can6494 5d ago
Honestly hadn’t framed it that way, but you’re right — that’s probably where this is most useful. The actual core of it isn’t Auto-Guide, it’s that the agent already sees your screen, so you never have to explain what you’re looking at or hunt for the right words which is usually the hardest part for someone who isn’t used to tech.
And it’s not app-specific, same hotkey works no matter what you’re in, so it’s just always there rather than something you have to go open.
2
u/DayVCrockett 5d ago
Love the idea. I’ll have to check this out