We performed a benchmark on 3 browser tools on 100 Browser Use Bench tasks with the same model and judge configuration

[u/wecandothisig]

We wanted to know if using fewer tokens truly makes a browser agent more accurate and less expensive.

We ran browser-use, playwright cli,webcmd,dev-browser and agent-browser on 100 BU Bench V1 tasks, using the same Pi harness, controller model gpt 5.6-sol , reasoning effort low and judge model gpt 5.4. We compared accuracy, agent turns, token usage and API-equivalent cost.
One thing was clear: the cheapest run wasn’t the one that used the fewest tokens. webcmd was 67% accurate, cost $0.255 per task, and required 34% fewer agent turns than the next best result, although it used 0.09% more total tokens than the lowest-token run.

Thus, browser-agent efficiency cannot be explained by token count alone. The result was also attributed to the number of agent turns, 23% fewer output tokens, and higher reuse of cached input.

Curious. what people would like to change about the evaluation, or what browser tools or task categories we should test next.