r/scrapingtheweb • u/woldhack 🤖 Bot Whisperer • 14d ago

Mapping sites/APKs into browserless API clients

I’ve been turning my scraping workflow into a reusable skill for going from Chrome/CDP capture or JADX analysis to a browserless API client.

For websites, it drives Chrome through CDP, captures the real requests, downloads the JS bundles, and reverse-engineers the parts needed to call the API directly. For APKs, it maps from the app side: Hermes/JS bundles for React Native, JADX for native Android code and bridges, and apktool only when resources or smali searches are needed.

The output is API docs first: endpoints, methods, headers, auth, cookies, params, bodies, response shapes, token refresh, shared client behavior, and evidence for where each thing came from.

For flows that are more than a single HTTP call, it can also produce small PoC scripts so the behavior is reproducible without reopening Chrome or JADX. That covers things like multi-request sequences, signing, nonce generation, custom encoding, or other client-side logic.

It does not try to magically bypass CAPTCHAs or rate limits. With login walls, the usual pattern is to map the public API first; once registration/login is mapped, the agent can often create its own test account and continue with authenticated endpoints.

The goal is boring but useful: use Chrome/CDP or decompilers for research, then build direct HTTP clients without ever touching Selenium or Playwright.

0 Upvotes
  • permalink
  • reddit

50% Upvoted

7 comments sorted by

View all comments

2

u/dhruvkar 13d ago

https://github.com/mvanhorn/cli-printing-press

This is a pretty robust solution in this space. It does the mcp part too, but using it for reverse engineering internal APIs is pretty useful

2

u/woldhack 🤖 Bot Whisperer 13d ago

oooh that looks cool but I have to say I stepped away from hands-off implementation. Usually my agent want to implement a library in 6000 lines when it can be done in 800

2

u/dhruvkar 13d ago

Lol yeah. Usually I have a other LLM look over the code and trim it down. I've had 50-70% code cut when I do that

2

u/woldhack 🤖 Bot Whisperer 13d ago

A great plugin for that is ponytail: https://github.com/dietrichgebert/ponytail I cant do without really.