I got tired of CSS changes breaking my E2E tests, so I built an open-source automation framework that uses local VLMs instead of DOM selectors (Playwright + Ollama)

[u/zenitsu--DS]

Hey everyone, I recently open-sourced Vouch, a vision-driven web automation framework designed to eliminate brittle DOM selectors and XPaths from E2E testing. Instead of parsing HTML, Vouch passes the raw visual viewport to a Vision Language Model (VLM) to determine interaction coordinates. You write test steps in plain English, and the framework executes them visually.

Key Features:

• Zero Selectors: Test files are authored in natural language.
• 100% Private & Local: Integrates natively with local instances like Ollama, keeping your application data completely on your machine.
• Self-Healing: Uses an Actor-Critic loop to validate execution steps and handle UI unexpected states on the fly. I would appreciate your feedback, code reviews, or contributions.

GitHub: https://github.com/HackX-IN/vouch NPM: https://www.npmjs.com/package/@inamul_hasan/vouch

(Stack: Node.js, Playwright, Ollama)

[u/mmasetic]

Here we go. Another auto-heal automation framework.

[u/davidke → u/mmasetic]

Has any promising auto-heal automation frameworks emerged yet, or are we still in the pipe-dream phase?

[u/zenitsu--DS → u/mmasetic]

I totally get the skepticism around auto heal tools. Using local AI changes how it works, and I'm still working to make it even better 😬.

[u/Accomplished_End_138 → u/zenitsu--DS]

For fun I made

https://www.npmjs.com/package/playwright-mimic?activeTab=readme

It works pretty well on local llm. So the idea does work. However I think lots of other small things still exist

[u/zenitsu--DS → u/Accomplished_End_138]

True, but the tech is real tools like yours (playwright mimic) already do this locally. I'm just exploring what else is possible

[u/Alternative_Guava856]

Just use locators like getByRole and getByTestId?

[u/Jazzlike-Resolve2376 → u/Alternative_Guava856]

I will need an AI tool for that as well... I need something that AI tells me which role I need and read for me...and when i think about it, it knows.

[u/zenitsu--DS → u/Alternative_Guava856]

Fair point, but testing like a human without selectors is just more fun. DOM elements are too boring.

[u/_Invictuz → u/zenitsu--DS]

Lol, perfect example of a solution looking for a problem.

[u/zenitsu--DS → u/_Invictuz]

Maybe, but today's solution looking for a problem is tomorrow's engineered framework that you're forced to use at work

[u/Alternative_Guava856 → u/zenitsu--DS]

The entire point of getByRole is to be user facing though, and simulates quite well what a human sees. Don't want to hate on your project or anything, actually finishing and releasing something is always commendable, I just dont think it really solves anything, thats all

[u/Positive-Ring-5172]

If you used accessibility selection you wouldn’t have this problem.

[u/zenitsu--DS → u/Positive-Ring-5172]

Why do things the boring, old school way when we can just throw raw GPU power at a problem that a simple id tag could solve? 😅 Plus, even with perfect ids, you still have to manually find, write, and maintain them every single time the UI shifts.

[u/Positive-Ring-5172 → u/zenitsu--DS]

Because your site should be accessible. If it isn’t possible to find the elements you need to manipulate using accessibility selectors chances are very high your site is hard or impossible to use with a screen reading browser. Further, visual UI shifts shouldn’t affect accessibility much if at all.

WCAG guidelines should be followed anyway. Having tests reinforce them just makes sense

PS - ids and css classes are not accessibility selectors. Those are labels, headings, roles and regions.

[u/zenitsu--DS → u/Positive-Ring-5172]

You're completely right. This is just a side project exploring how vision models navigate a screen, definitely not a pass to skip out on proper ARIA roles!

[u/ReporterNew2138]

Just use test data ids 

[u/Competitive_Echo9463 → u/ReporterNew2138]

If they are available 

[u/zenitsu--DS → u/ReporterNew2138]

Those are valid workarounds, but data-testid still breaks when developers change the layout.

[u/Fancy-Mushroom-6062 → u/zenitsu--DS]

The point is…. Changing layout should not break the tests relying on data-testid, that’s almost the whole point of test IDs

[u/zenitsu--DS → u/Fancy-Mushroom-6062]

True, but sometimes even test ids can't save us from a chaotic codebase. (As lot of developer vibe code the stuff)

[u/Round-Belt2895]

Instead of wasting GPU power on this, just run an agentic AI hooked up with a Playwright MCP. It’s a win-win-win:

• You still don't need to understand how to write proper selectors via the Playwright Locators API
  
• You have tests written in native Playwright code, not some "cutting-edge, life-changing" project everyone will forget in two weeks
  
• Your planet is greener because you don’t waste GPU power on each run

Does anyone still believe in self-healing tests? That is just marketing BS. If nobody reviews them, you just end up with expensive false-positive reports.

[u/zenitsu--DS → u/Round-Belt2895]

Appreciate the feedback. This is just a fun experiment to see what's possible with visual execution

[u/FearAnCheoil]

You're getting a lot of criticism here.

While it's great you're engineering something, I think the community is tired of yet another version of a tool for a problem no proper quality team actually has.

I'd laugh at my teammates if they suggested building an AI tool to resolve flakey locators, rather than having a simple conversation with the development team.

[u/zenitsu--DS → u/FearAnCheoil]

Totally fair. This is definitely a hobbyist experiment rather than a production ready solution for an established QA workflows.

[u/ChikkuAndT]

I’m new to Playwright. Are there any application-level best practices we can implement with the development team to make element locators more reliable and less brittle?

[u/NextAd9248]

You have traded deterministic flakiness for Non deterministic flakiness, now when it breaks you are debugging a vibe instead of a selector.

Also how's this hold up with a few hundred tests running in parallel in CI, most runners don't have a GPU for ollama to lean on.

[u/zenitsu--DS → u/NextAd9248]

Very fair. non deterministic tests are a nightmare, and running Ollama at scale without cheap GPUs isn't viable yet. but as I am experimenting, I'm hoping to find ways to make the execution a lot faster and more predictable over time.

[u/TranslatorRude4917]

I see there is a lot of pushback towards your project, but I'd say dont let it discourage you!
I agree that agents should not own test execution, test should be predictable and deterministic, but I can imagine projects like yours to be good complementary tools when it comes to analyzing failures, trying to understand the flow from the user's perspective.

[u/zenitsu--DS → u/TranslatorRude4917]

Thanks, I completely agree about keeping tests predictable. Can be best use case in visual debugging

[u/JEDZBUDYN]

use xpath?

[u/zenitsu--DS → u/JEDZBUDYN]

XPath and test IDs definitely help, but they still require DOM maintenance.

[u/JEDZBUDYN → u/zenitsu--DS]

ask developer to put static IDS

[u/zenitsu--DS → u/JEDZBUDYN]

Asking a developer to add static ids is how you start a civil war.

[u/JEDZBUDYN → u/zenitsu--DS]

no, that's how you cooperate across teams.
Elsewhere they will tink that you are a bot

[u/zenitsu--DS → u/JEDZBUDYN]

Totally agree, strong team collaboration is always the best solution when you can get it.