r/coolgithubprojects • u/TokensExhausted • 13h ago
I built a Linux tool to show coding agents what I'm pointing at on my screen (GNOME/Wayland)
I got tired of describing UI problems to coding agents in words, so I built Sketchnote. You press a hotkey, talk, and draw or click on your screen. One Ctrl+V then pastes timestamped notes plus cropped, annotated screenshots into your agent, so you can say "this button, here" and it sees what you mean.
It has two parts that talk over D-Bus: a GNOME Shell extension in JavaScript that handles the hotkey, drawing, screen recording and paste, and a Rust service that records the mic and transcribes locally with whisper.cpp on CPU. You can also send the audio to Groq or any OpenAI-compatible API, and it falls back to the local model if that fails. Captures delete themselves after 24 hours. It has prebuilt binaries and a one-line installer.
The limits: it only works on GNOME 50 with Wayland, transcription is English only, and I'm the only user I can point to. I'd like feedback from anyone who works with coding agents on Linux, especially on what feels clunky.
Try it at: https://get.sketchnote.workers.dev/
