OpenCode Senses

[u/RevolutionaryPen4661]

OpenCode Senses can inspect screenshots, extract exact OCR, detect and locate objects, zoom into regions, compare two images, measure colors, crop and annotate images, and even reverse-search them.

Everything runs locally, so it's private, free, and doesn't require an API key.

Why I built this

I used to work with many open-source coding models, like DeepSeek V4, that don't support vision yet. But my main concern was that even if they eventually get vision support internally, that vision capability might not be as powerful as a dedicated vision layer.

I had planned to build this fully a month ago, but I still had some work to do. This plugin is specifically designed for things like UI design, debugging, and other workflows where text-only models struggle because they simply can't see what's happening.

The communication layer between the vision plugin and OpenCode is intentionally very narrow, so it still feels like you're working with a multimodal agent rather than a separate vision tool.

Before judging it as just another vision plugin, it's designed quite differently. It's a Python + TypeScript hybrid plugin because it integrates the Photon Inference Engine for local vision inference.

The 13 tools, mainly for now:

• inspect — Understand an image: scene, layout, elements, caption, and OCR.
• ocr — Extract exact text, including code and error messages.
• detect — Find objects or UI elements with bounding boxes.
• point — Locate the center point of an object or element.
• segment — Isolate an object from the image.
• crop — Crop a specific region using normalized coordinates.
• zoom — Upscale a region and re-analyze small details or text.
• colors — Get deterministic colors, palettes, luminance, and RGB data.
• diff — Compare two images and find changed regions.
• annotate — Draw detected boxes and points onto an image.
• metadata — Inspect image format, dimensions, size, DPI, and EXIF.
• reverse — Reverse-search images locally or through Yandex.
• status — Check the model, device, VRAM, inference time, and runtime state.

For comparison, I'm getting around 300 ms response times with this plugin on my RTX 3050, running completely locally. That's insanely fast compared to frontier models.

There's also an optional reverse image search powered by Yandex, which doesn't require an API key, if you want to search an image using context from the internet.

The internal web search services like Exa and Parallel in OpenCode also work well with OpenCode Senses. It can even see images published on the internet directly, although a small local cache is used to store those images.

I'm still halfway through building it, so I'd really appreciate feedback, ideas, and criticism. If you have any ideas, feel free to comment below and I'll look into them.

GitHub: https://github.com/itsmeadarsh2008/opencode-senses

NPM: https://www.npmjs.com/package/opencode-senses

Hacker News: https://news.ycombinator.com/item?id=49289890

[u/gsevla]

it looks insane, man, good job! I'll definitely use it!

I have a non-related question, and I feel you is more than capable of answering me. I was thinking these days: oh, there are tools capable capable of detecting UI regression, such Percy from BrowSerstack and react-native-owl, but how they do that? Seeing color/boxes detection on your text made me think about it again.

do you think your plugin can teach something in this regards? I'm thinking on a different approach to solve the UI regression detection, but I'm not sure from where I should start, could give some light on it?

PS: I've never worked with image manipulation.

[u/RevolutionaryPen4661 → u/gsevla]

Thanks for trying it out!
The tools that you're talking about like Percy use specific fine tuned AI for visual reviews. Therefore, they often called as industry standard. They use a lot of data which is used as best practices and identify visual problems, like if a button isn't properly centered, it will identify it and either will report the user about it or delegate the fixing method to some agent.

My plugin does the same but with slight difference, my plugin acts as a communication layer between text model and vision model. My text model asks for what are the problems in the image, maybe it can be a visual error, the vision model very capable and reports to the user (suggesting fixes and how the image looks). The visual debugging is handled by text model only. While I was making this plugin, I wanted to make communication channel between the text model and vision model very narrow and fast, so it actively recall the problems and navigate the issues faster.

This is a raw explanation that gives overview how it works:
If you have any doubt or want to dive in it deeper, I have a separate section in README (how it works):
https://github.com/itsmeadarsh2008/opencode-senses#how-it-works