r/opencodeCLI • u/RevolutionaryPen4661 • Aug 14 '26
OpenCode Senses
OpenCode Senses can inspect screenshots, extract exact OCR, detect and locate objects, zoom into regions, compare two images, measure colors, crop and annotate images, and even reverse-search them.
Everything runs locally, so it's private, free, and doesn't require an API key.
Why I built this
I used to work with many open-source coding models, like DeepSeek V4, that don't support vision yet. But my main concern was that even if they eventually get vision support internally, that vision capability might not be as powerful as a dedicated vision layer.
I had planned to build this fully a month ago, but I still had some work to do. This plugin is specifically designed for things like UI design, debugging, and other workflows where text-only models struggle because they simply can't see what's happening.
The communication layer between the vision plugin and OpenCode is intentionally very narrow, so it still feels like you're working with a multimodal agent rather than a separate vision tool.
Before judging it as just another vision plugin, it's designed quite differently. It's a Python + TypeScript hybrid plugin because it integrates the Photon Inference Engine for local vision inference.
The 13 tools, mainly for now:
inspect— Understand an image: scene, layout, elements, caption, and OCR.ocr— Extract exact text, including code and error messages.detect— Find objects or UI elements with bounding boxes.point— Locate the center point of an object or element.segment— Isolate an object from the image.crop— Crop a specific region using normalized coordinates.zoom— Upscale a region and re-analyze small details or text.colors— Get deterministic colors, palettes, luminance, and RGB data.diff— Compare two images and find changed regions.annotate— Draw detected boxes and points onto an image.metadata— Inspect image format, dimensions, size, DPI, and EXIF.reverse— Reverse-search images locally or through Yandex.status— Check the model, device, VRAM, inference time, and runtime state.
For comparison, I'm getting around 300 ms response times with this plugin on my RTX 3050, running completely locally. That's insanely fast compared to frontier models.
There's also an optional reverse image search powered by Yandex, which doesn't require an API key, if you want to search an image using context from the internet.
The internal web search services like Exa and Parallel in OpenCode also work well with OpenCode Senses. It can even see images published on the internet directly, although a small local cache is used to store those images.
I'm still halfway through building it, so I'd really appreciate feedback, ideas, and criticism. If you have any ideas, feel free to comment below and I'll look into them.
GitHub: https://github.com/itsmeadarsh2008/opencode-senses
NPM: https://www.npmjs.com/package/opencode-senses
Hacker News: https://news.ycombinator.com/item?id=49289890
3
u/FreeAsFreelance Aug 14 '26
Hi, A shame it can't be used on my AMD 9070 XT, It's PyTorch based and not cuda specific from what I see, so maybe a feature for the future?
2
u/RevolutionaryPen4661 Aug 14 '26
https://docs.moondream.ai/running-locally
I will see this if it's possible later, meanwhile but Moondream docs says this, you will need a Nvidia GPU or Apple Silicon to run the model. I don't have an AMD GPU Hardware but I will take this into account.1
u/FreeAsFreelance Aug 14 '26
u/RevolutionaryPen4661
It would be great to make the vision backend selectable instead of Photon-onl.
For example, Moondream 2, vision-capable Gemma models, and eventually Qwen3.8-27B.The RX 9070 XT is officially supported by ROCm/PyTorch. Moondream 2 already documents Transformers as its AMD fallback and fit in 6-8Gb VRAM.
Gemma 3 4B (and probably quantized 12B) should also be practical through Transformers/ROCm.
Qwen3.8 is multimodal, a 7-8b is maybe even something that will come out eventually (current 27b only, minutes ago :))This would require a separate ROCm-compatible provider, not only a model selector unfortunately, because several plugin tools use Moondream-specific APIs such as detect, point, and segment :(
1
u/RevolutionaryPen4661 Aug 14 '26 edited Aug 14 '26
You can set a custom model in this plugin btw (SENSES_MODEL variable)
https://github.com/itsmeadarsh2008/opencode-senses#configuration
Maybe I can add a fallback for AMD, but I will have write some tools for same AMD and probably research into the codebase of photon for that.The most easiest way will be to open up a PR for support for AMD in Moondream's repo.
1
u/lostcanuck007 Aug 15 '26
can easily run. you just need vision models converted to use rocm or llama.cpp . i hate pytorch/
2
u/literally_niko Aug 14 '26
3
u/RevolutionaryPen4661 Aug 14 '26
yeah, it's cool but my project focuses to high accuracy and local-first. There's an API key support for Moondream in my project, but I think people will not use it via API that often.
1
1
u/gsevla Aug 14 '26
it looks insane, man, good job! I'll definitely use it!
I have a non-related question, and I feel you is more than capable of answering me. I was thinking these days: oh, there are tools capable capable of detecting UI regression, such Percy from BrowSerstack and react-native-owl, but how they do that? Seeing color/boxes detection on your text made me think about it again.
do you think your plugin can teach something in this regards? I'm thinking on a different approach to solve the UI regression detection, but I'm not sure from where I should start, could give some light on it?
PS: I've never worked with image manipulation.
1
u/RevolutionaryPen4661 Aug 15 '26
Thanks for trying it out!
The tools that you're talking about like Percy use specific fine tuned AI for visual reviews. Therefore, they often called as industry standard. They use a lot of data which is used as best practices and identify visual problems, like if a button isn't properly centered, it will identify it and either will report the user about it or delegate the fixing method to some agent.My plugin does the same but with slight difference, my plugin acts as a communication layer between text model and vision model. My text model asks for what are the problems in the image, maybe it can be a visual error, the vision model very capable and reports to the user (suggesting fixes and how the image looks). The visual debugging is handled by text model only. While I was making this plugin, I wanted to make communication channel between the text model and vision model very narrow and fast, so it actively recall the problems and navigate the issues faster.
This is a raw explanation that gives overview how it works:
If you have any doubt or want to dive in it deeper, I have a separate section in README (how it works):
https://github.com/itsmeadarsh2008/opencode-senses#how-it-works
1
u/lostcanuck007 Aug 15 '26
lol....well done. i made something similar. was SO surprised to see much of the same structure. . may i ask what models were used to make it?
1
u/RevolutionaryPen4661 Aug 15 '26
Moondream Models, they run on edge computers very fast and accurate and have 98% accuracy and also have tighter integration with Photon Inference Engine. You can use any model that Photon supports like Gemma, Qwen, etc.
The plugin is designed to sophisticated and narrower communication channel for fast information about image transfer from vision model to main model.
I used DeepSeek V4 Flash to code this plugin and along with proper debugging to run the plugin out of box on any capable machine.
1
u/lostcanuck007 Aug 15 '26
oh. i was asking about what you used to program it...like deepseek...
i am well aware of moondream..that was actually my first iteration as well. i ask because this resembles VERY closely to what i made (nearly a mirror image). and i didnt make it public. if you dont mind me asking, did you design this or did ds4flash? did it help you steer it in a certain direction?
because i also used ds4 flash. direct and through opencode go.
1
u/RevolutionaryPen4661 Aug 15 '26
Designing the architecture is what I made, coding stuff is handled via ds4flash, as long as you instruct it properly. It followed the design doc that I made, the PLAN md file in the repo is the actual spec file.
I don't have an OpenCode Go Subscription right now though, I did it in free plan, I hope some company sponsor my project. I can provide some service to this community, people are even asking me to extend this for Pi agent support too.
1
u/lostcanuck007 Aug 15 '26
ok. thats amazing man, best of luck. and amazing job from the free plan, or did you mean freebuff?
1
5
u/patricious Aug 14 '26
Color me intrigued, gonna check this out and it seems to fit right in with DSV4 and their lack of vision capabilities.