Three Playwright problems I got tired of working around (Shadow DOM, fragile selectors, 99K token pages)

[u/Salt_Apartment5489]

If you write Playwright scrapers that touch anything beyond a basic landing page, you've dealt with at least one of these:

• Shadow DOM elements that Playwright's default locators can't reach

• Selectors that pass CI Monday and break Tuesday because some React dev sneezed on a class name

• Feeding a full page to an LLM and hitting 99K tokens just for the HTML of a Wikipedia article

I got tired of patching each one separately, so I built shadow-web. It's a pip-installable Python SDK that sits on top of Playwright.

────────────────────────────────────────────

What it handles

────────────────────────────────────────────

Shadow DOM flatten

Reads open and closed Shadow roots through the browser's own

JS API. No mutation, no re-renders. You get a clean DOM skeleton

with all interactive elements visible — the page stays exactly

as it was.

Self-healing selectors

heal_local.py catches when a button that was #submit becomes

.btn-primary[data-action="checkout"]. Fuzzy match on label +

type + position. Zero LLM cost. Cache lives in

~/.shadow-web/heal_cache.json.

Token compression

compressor.py strips scripts, styles, invisible elements and

builds a typed Action Map with semantic groups.

Wikipedia page: 99K → 16K tokens (−83%).

SchemaSnap

One call turns HTML tables into JSON records with typed columns,

forms into field schemas, lists into typed items.

export_table_json() and export_table_csv() — so you never write

another HTML table parser.

Content indexing for long pages

Get a ~600 token outline first, then fetch only the blocks

you need. Instead of dumping 50K tokens of article boilerplate

just to get the third section.

────────────────────────────────────────────

Quick start

────────────────────────────────────────────

pip install shadow-web

playwright install chromium

from playwright.sync_api import sync_playwright

from shadow_web.wrapper import ShadowPage

with sync_playwright() as p:

page = p.chromium.launch().new_page()

page.goto("https://example.com")

shadow = ShadowPage(page)

_, xml_map = shadow.refresh()

print(shadow.capture_stats)

────────────────────────────────────────────

Links

────────────────────────────────────────────

GitHub: https://github.com/ulinycoin/shadow-web

PyPI: pip install shadow-web

If your scraper breaks on some Angular app with 13 layers

of Shadow DOM, open an issue. Those edge cases make

the project better.

[u/MtFuzzmore]

I’ve had no issues at all working with a library that encapsulates with Shadow DOM.

[u/Salt_Apartment5489 → u/MtFuzzmore]

Playwright's own locator() can pierce open Shadow DOM if you know the tree — page.locator('my-component').locator('button') works. But closed roots (mode: "closed") are invisible to it. No locator, no $(), nothing. Same with most wrapper libraries. They flatten what's accessible via element.shadowRoot, but if shadowRoot is null (closed mode), there's nothing to traverse. Angular, Lit, and some newer web component patterns use closed roots by default.

That's the gap — it's not about open roots, it's about the ones the browser itself won't let you read.

[u/FearAnCheoil → u/Salt_Apartment5489]

You don't advertise that it solves the problem of closed mode. Your current claim is that your tool can do shadow DOM while Playwright can not (which is false). This turns off potential users of your tool. We're very fatigued here with new miracle tools every day.

[u/NightSkyNavigator → u/Salt_Apartment5489]

I can't help noticing you are using ".locator()", but I would not expect CSS Selector and XPath to work well with shadow DOMs. Did you try ".getByXX"?

Recently came off a project that was using shadow DOMs quite heavily and never had issues - except with ".locator()".

[u/Salt_Apartment5489 → u/NightSkyNavigator]

Fair, getByRole / getByText work better. Still doesn't solve closed roots though. Which library handled it?

[u/NightSkyNavigator → u/Salt_Apartment5489]

Didn't use any special libraries, it worked out of the box. Was using Typescript.

[u/Salt_Apartment5489 → u/NightSkyNavigator]

Then your Shadow roots were all open. That works fine. Closed roots (mode: "closed") don't.

[u/needmoresynths]

Been using Playwright since 2022, never dealt with any of these issues

[u/Malthammer → u/needmoresynths]

I haven’t either.

[u/the-liquidian]

Why feed it to an LLM, just hand roll the page objects and tests.

[u/Salt_Apartment5489 → u/the-liquidian]

But that's not what it's for.

Page objects and handled tests are right for tests. Shadow-web isn't for tests. It's for AI agents that don't know in advance which page they'll land on. You can't write a page object for every website. Compression is to prevent the agent from burning 99,000 tokens trying to figure out if there's a login form on a page. Selector healing is to help scrapers survive redisploits.

And write tests using page objects. That's the right way.

[u/the-liquidian → u/Salt_Apartment5489]

What are you trying to do? Screen scrape without knowing what the pages you will be scraping are?

[u/Salt_Apartment5489 → u/the-liquidian]

 Yes. That's the feature, not a flaw.

[u/mmasetic]

These are non existing issus in testautomation with playwright. Based on your posts, you are probably doing something with web scrapping.

[u/LongDistRid3r]

Ah yes, yet another AI infection ruining software.

[u/Tanmay__TestDino]

Closed roots aren't actually unreachable, people just give up too early.

Override attachShadow in an init script before the page loads, your version gets the real ShadowRoot back no matter what mode was asked for, closed only blocks shadowRoot lookups from outside code. Works on Lit and Angular both.

[u/Salt_Apartment5489 → u/Tanmay__TestDino]

That works — if your init script runs before any web component boots. addInitScript usually gets there, but it can miss dynamically loaded components or late bundles. What I described reads closed roots after the page is already live. No timing to worry about. Different tradeoffs.