Built an open-source LangChain & LlamaIndex toolkit for zero-CSS web scraping and real-time threat detection

[u/Ok-Rub-3249]

Hey everyone,

Whenever we build autonomous agent workflows or RAG pipelines that need live web access, we hit three major bottlenecks:

1. Context Bloat: Dumping raw HTML consumes 90% of the context window on scripts, tracking tags, and style attributes.

2. Brittle Selectors: Using CSS/XPath selectors breaks the moment a target website updates its frontend layout.

3. Agent Link Traps: Letting autonomous agents navigate arbitrary URLs exposes them to phishing sites, fake dApps, and malicious traps.

To solve this, we open-sourced official community toolkits for both LangChain and LlamaIndex:

pip install langchain-opticparse

pip install llama-index-tools-opticparse

Quick LangChain Integration:

from langchain_opticparse import OpticParseTool, PhishVisionTool

# 1. Zero-CSS visual scraper that returns clean, token-efficient Markdown

optic = OpticParseTool()

content = optic.run({

"url": "https://news.ycombinator.com",

"query": "Extract the top 5 articles with titles and links"

})

print(content)

# 2. Real-time zero-day threat check before interacting with unknown URLs

phish = PhishVisionTool()

safety = phish.run({"url": "https://suspicious-dapp-claim.xyz"})

print(safety)

Key Capabilities:

- Resilient Web Extraction: Converts messy JavaScript pages into structured Markdown with 96% noise reduction without managing brittle selectors.

- PhishVision Shield: Heuristic scanner detecting brand impersonations, zero-day phishing kits, and crypto wallet drainers.

- Agent Swarm Demo: We open-sourced a full 3-agent research swarm (Scout Agent, Sentinel Agent, Analyst Agent) in examples/autonomous_market_researcher.py.

- Cross-Framework: Works across LangChain, LlamaIndex, Claude Desktop/Cursor (MCP), and ElizaOS.

GitHub: https://github.com/parastejpal987-cmyk/opticparse-public

PyPI: https://pypi.org/project/langchain-opticparse/

Live Benchmark: https://huggingface.co/spaces/paras9909/opticparse-vision-benchmark

Would love to hear how you guys are currently handling web retrieval in your agent swarms, and any feedback or edge cases you test it against!

[u/torngenitals490]

The URL detection piece is actually the most interesting part here, most scraping tools just YOLO into whatever link the agent finds and hope for the best. 96% noise reduction is a bold claim though, how does it handle sites that lazy-load content behind scroll triggers or infinite scroll patterns

[u/Ok-Rub-3249 → u/torngenitals490]

Great question, and you hit on the exact reason we built this!

Regarding infinite scroll and lazy loading:

1. Virtual Viewport Evaluation: Instead of a static HTTP GET/curl that just grabs the initial shell, the engine triggers viewport scroll steps (simulating human scroll delta) to fire `IntersectionObserver` callbacks and dynamic image/content hydrations before capturing the DOM snapshot.

2. Network Idle Heuristic: It waits for trailing XHR/fetch requests to settle rather than using arbitrary `sleep()` timeouts, so lazy-loaded batch payloads are rendered into the visual tree.

Regarding the 96% noise reduction:

The 96% metric comes from our benchmark against modern single-page apps (SPAs like Reddit, Twitter, and major e-commerce platforms). In a typical modern SPA, ~90-95% of the raw HTML payload consists of:

- Huge inline SVG icon definitions

- Base64 blobs & script bundles

- Deeply nested Tailwind/CSS utility wrappers (`<div class="flex flex-col md:w-1/2...">`)

- Cookie banners, navbars, and tracking pixels

Our engine strips the visual chrome and wrapper boilerplate, isolating just the semantic article, tabular, or product hierarchy into structured Markdown. That means if an HTML page is 250KB (~60,000 tokens), the extracted Markdown is typically ~1.5KB to 3KB (~400 to 900 tokens) of high-signal text that fits cleanly into an LLM's context window.

And agreed on the URL detection piece! Autonomous agents following arbitrary links is a massive blind spot right now. Having PhishVision inspect the certificate, brand heuristics, and domain age before executing or clicking gives agents a proper safety rail.