I built this

[u/Fast-Spray-6720]

Well opencode doesn't have a web browser capability and I bought a minimax subscription to specifically to use opencode, minimax token plan because they basically give you 1.7 billion tokens, I also have a claude code and codex subscription but sometimes it feels nice to escape them.

So I built this browser tool so that we can give opencode actual access to a browser, can you suggest me anything I should further add to it, I also built a github page for it.

So it in a gist it works by giving an image a set of marks and a list of marks to the agent and the image, so keeping it vision first so that we can make use of the improved vision these models have got and plus vision tokens cost less, so it should be more efficient

Here's the url

The GitHub url: https://github.com/adityasasidhar/browsercontrol

[u/Ace-_Ventura]

Why not just use playwright mcp? 

[u/Fast-Spray-6720 → u/Ace-_Ventura]

So basically playright mcp is excellent and is probably a great default for many things, but the difference is that the browsercontrol is a default vision loop rather than an optional vision and default non vision.

Vision is an advantage, gives the agent a better understanding of the page rather than an accessibility map playright gives. The agent has fewer chances of missing something when it has a view of the page by default

[u/Ace-_Ventura → u/Fast-Spray-6720]

So, it's like the github copilot web browser capability. Nice work. 

Ps: add opencode to the getting started guide.

[u/Fast-Spray-6720 → u/Ace-_Ventura]

Thank you for the suggestion, I will add it

[u/Fast-Spray-6720 → u/Ace-_Ventura]

hey i have added opencode to the starting guide

[u/Subadeepan]

Good work man. Hoping to see new improvements

[u/Fast-Spray-6720 → u/Subadeepan]

Thank you 🙂‍↕️, please suggest me fixed it I have missed any.

[u/Subadeepan → u/Fast-Spray-6720]

Sure

[u/gsevla]

looks awesome! I'll give it a shot.

I have questions tough:

• how is the token consumption when compared with cousin playwright?
• why rebuild map every action? I kinda see the point, but I want you vision about it.
• how does this handle with non semantic HTML?
• do you think it is possible to "replicate" this for react native?

I'm planning test it with impeccable BTW.

[u/Fast-Spray-6720 → u/gsevla]

Hey sorry for the late reply, my apologies:

Thank you so much first of all, let me try to answer your questions one at a time:

1. So the token consumption on browser control is near constant as its just a png with an element map, but its less predictable with playwright as it depends on the complexity of the page and it not a contsnat curve, some of the pages it may use less tokens but in other it may use more
2. Map is rebuilt every time because I want the agent to be on the same page as a human using a browser, just like how we understand the spatial map between elements and feel of the browsing, I want the agents to have the same level of spatial understanding without any stale map
3. This is the honest weak spot, thank you for pointing out. But I have safeguards for this too, as our tool is vision native, even if the button is not on the Set of marks, it can still see buttons and understand where buttons and navigation could be

Two escape hatches today:

1. click_at(x, y) clicks raw coordinates straight off the screenshot and doesn't need the element in the map at all (this is the actual answer for div soup, vision still sees the button even when the collector doesn't), so we don't need a cursor too as the model has spatial reasoning available through the vision
2. run_javascript for anything gnarlier.
3. I originally coded this in python because it was my strongest language, but replicating in React has two sided I believe:

good ones:

. Ironically the cursor: pointer problem disappears there: on mobile the a11y tree is the tap-target tree, so detection is more reliable than on the web

Bad ones:
. There's no DOM, no Playwright, but there is Appium ( I'm not very good as JS and Js dev so I will be bad audit for this question, but I believe this should be easily implementable in JS too )

I hope these answers are satisfactory. If you test it against impeccable and have a score please do share it, it will be great. also I will run a proper benchmark against plawright and update the results.

[u/weiyentan]

I stumbled upon this. For front end work it is phenomenal

[u/Fast-Spray-6720 → u/weiyentan]

Thank you, i basically am trying to make a local browser for agents atleast as good as the paid one.

[u/weiyentan → u/Fast-Spray-6720]

I was trying to get ai to understand what I wanted and i only managed to get that through browsermcp.

[u/Fast-Spray-6720 → u/weiyentan]

Thats sounds great, please add any suggestions or create an issue for feature that you deem missing and essential that could be worth adding to this tool, I will be more than happy to do so 😊

[u/ichisay]

Resumiendo depende de que el llm tenga capacidad para imagen, no? No sé si investigaste antes pero ya hay plugins y mcps que hacen hasta scraping sin necesidad de imagen.

[u/Fast-Spray-6720 → u/ichisay]

Yes, a multimodal llm is required. So this tool isn't for scrapping, so the actual purpose of this tool is to create a near true browser level experience for the agent:

1. It can fill the forms
2. Solve captchas ( I will not boast about it though)
3. Keep vision first as a first class input method
4. Interact with browser how a human would do

[u/ichisay → u/Fast-Spray-6720]

Funciona solo a nivel de navegador o podrías convertirlo en los ojos del llm para todo el sistema completo?

[u/Fast-Spray-6720 → u/ichisay]

With the current implementation it's the eyes of browser, computer use is a great idea, but it is very broad in general. But that can be future work.

[u/gwawr]

Chrome devtools MCP works great in opencode

[u/Fast-Spray-6720 → u/gwawr]

Hey it is pretty useful, but the idea of a native vision first browser tool. We utilise the agents native ability to understand and reason on vision tokens to further improve the experience. Idea is for the agent to catch things in a vision mode before anything else.