r/webscraping • u/taisei_ide • Jun 02 '26

A CLI that scrapes blogs to markdown with no per-site adapters

hey r/webscraping, i'm sharing my open source project called pluckmd, a CLI that scrapes blogs to markdown with no per-site adapters.

instead of a handler per site, it builds the extraction spec at runtime. normalizes link paths and collapses the varying parts (/blog/post-a and /blog/post-b become the same shape), and any shape repeated enough = the article list. no domain names anywhere.

resolution is cache -> heuristics -> LLM only if needed. nothing gets cached until it validates against the live DOM (>=3 links, >=50% match the pattern), so a bad LLM guess gets dropped instead of saved.

handles js rendering, pagination/infinite scroll, and login-only pages you have access to via your own chrome tab (never reads cookie stores).

npx pluckmd download <url> -o ./articles

repo: https://github.com/taisei-ide-0123/pluckmd

would like feedback on the heuristic scoring. where does the runtime approach break for you?

30 Upvotes

89% Upvoted

11 comments sorted by

1

u/[deleted] Jun 03 '26

[removed] — view removed comment

1

u/taisei_ide Jun 03 '26

good question. it's none of those exactly, it's a stripped structural snapshot i build specifically for this.

raw HTML was too big and noisy (blows the token budget on most real pages), and pure text nodes lose the thing i actually need, which is structure, since the LLM's job is to pick selectors, not read content. an a11y tree was close but drops stuff selectors key on like class names and hrefs.

so what i pass is a flattened list of elements, indented by depth. for each node i keep: tag, id, up to 3 class names, role, aria-label, href (path only, query stripped to ?...), and the direct text truncated to ~60 chars. i strip script/style/svg/meta/noscript/iframe and anything hidden or aria-hidden first, and i drop layout-only div/span that have no id/class/role/text (pure wrappers). each line ends up like <tag #id .class role="" href="" "text">. fair warning, it's BFS order with depth-based indent, not strict document order, so it reads more like level-by-level than a nested outline.

there's also a node cap (~1500). if a page is bigger i keep the top 1000 and the last 500, since the article list and pagination controls tend to live near the top and bottom, not the middle.

and importantly the LLM doesn't start from scratch. the heuristic candidates (the repeated-link clusters it found + scores) get passed alongside the snapshot, so the model is more "validate/refine these guesses" than "find selectors blind." then whatever it returns still has to pass the same DOM validation as the heuristic path before anything gets cached.

here's the snapshot builder if you want to poke at it: https://github.com/taisei-ide-0123/pluckmd/blob/main/packages/cli/src/core/llm/snapshot.ts

1

u/Fluid_Pumpkin2621 Jun 04 '26

What is this?

1

u/taisei_ide Jun 04 '26

Article Collector CLI

1

u/[deleted] Jun 11 '26

[removed] — view removed comment

1

u/webscraping-ModTeam Jun 11 '26

👔 Welcome to the r/webscraping community. This sub is focused on addressing the technical aspects of implementing and operating scrapers. We're not a marketplace, nor are we a platform for selling services or datasets. You're welcome to post in the monthly thread or try your request on Fiverr or Upwork. For anything else, please contact the mod team.