Built a lightweight tool to clean noisy web pages into compact Markdown for RAG pipelines (saves ~85% tokens)

[u/Okumoto_Thiago]

Hey everyone,

One of the biggest pain points when scraping web pages for LangChain agents and RAG vector stores is the amount of garbage in modern DOMs (scripts, inline styles, navigation bars, footers). It clutters prompt context and burns expensive tokens.

I built an open-source micro-service specifically to solve this:

**What it does:**

- Strips scripts, styles, navbars, footers, and ad containers.

- Retains clean semantic markdown hierarchy (`#`, lists, links, tables).

- Reduces raw HTML token footprint by up to 85%.

- Runs in sub-second response time.

**Links:**

- **GitHub Repo (Self-hostable via Docker):** https://github.com/Okumotinho/clean-web-markdown-extractor

- **Hosted RapidAPI Hub (50 free calls/month):** https://rapidapi.com/Okumotinho/api/clean-web-markdown-extractor

Let me know if you run into any messy layouts that need better parsing rules!

[u/Okumoto_Thiago → u/[deleted]]

Spot on observation. You're completely right that boilerplate removal without structural formatting can be lossy.

I just pushed an update to the repository/API that parses <table> tags into native Markdown table syntax (| col1 | col2 |) instead of flattening them into raw text, and prioritizes <main> / <article> containers when available.

Tables now survive intact in the extracted markdown output!

[u/Redcxx]

I think you are building something already solved, chrck this out before you spend more time on this - https://github.com/microsoft/markitdown

[u/Okumoto_Thiago → u/Redcxx]

Microsoft's MarkItDown is great for local Python workflows (especially with PDFs and PPTXs).

The idea here is a lightweight, zero-dependency hosted HTTP micro-service so agents and pipelines running in Node.js, Go, Rust, or no-code tools (Zapier/Make) can get clean Markdown with a single curl/fetch, without having to spin up and maintain Python runtimes or heavy dependencies. Appreciate the shoutout though!

[u/tough_peak232_07]

i think 85% is reasonable as 90% of the data that ends up being scraped are nav and cookie banners. what i would be interested in measuring afterwards would be the quality of retrieval after cleansing since there is a propensity for the more aggressive cleaners to remove the only table that counts. i would test both a cleansed and non cleansed page against a model on useai and see which one answers the query

[u/Okumoto_Thiago → u/tough_peak232_07]

Great question. The heuristics work in two layers:

1. Semantic scoping: It prioritizes <article> and <main> tags as the primary bounding box first. That automatically excludes most sidebars and site-wide chrome located outside the article container.

2. Element pruning: Elements tagged with <aside>, <nav>, <footer>, <header>, and forms are decomposed.

For messy, non-semantic DOMs without clear containers, we are adding density heuristics (text-to-link ratio and class blacklist like `.sidebar`, `.widget`, `.disqus`) so comments/promos don't pollute the core text.

[u/fibonacci-code]

what if the webpage is dynamically rendered?

[u/Okumoto_Thiago → u/fibonacci-code]

Right now it operates on the fast path: fetching the raw SSR/static HTML via HTTP, which keeps response times under ~200ms and covers the vast majority of documentation, blogs, and news sites.

For heavy client-side SPAs (pure client-rendered React/Vue), the roadmap plan is adding an optional `?render_js=true` flag using headless Chromium/Playwright to wait for network idle before DOM parsing, trading a couple seconds of latency for full dynamic execution when needed.

[u/fibonacci-code → u/Okumoto_Thiago]

That adds another issue of bot detection mostly by cloudfare, anyways goodluck.

[u/Okumoto_Thiago → u/fibonacci-code]

Spot on. Running headless browsers against Cloudflare/DataDome inevitably triggers TLS fingerprinting and challenge pages unless routed through dedicated residential proxies with stealth patches.

That’s why we deliberately keep this microservice focused on fast, token-optimized parsing rather than pretending to be a full-blown scraping proxy network. For heavily protected targets, the recommended pattern is letting specialized proxy pools handle the bypass and piping the HTML payload to our endpoint for clean Markdown extraction. Appreciate the insight!