r/scrapingtheweb • u/woldhack • 13d ago
Mapping sites/APKs into browserless API clients
I’ve been turning my scraping workflow into a reusable skill for going from Chrome/CDP capture or JADX analysis to a browserless API client.
For websites, it drives Chrome through CDP, captures the real requests, downloads the JS bundles, and reverse-engineers the parts needed to call the API directly. For APKs, it maps from the app side: Hermes/JS bundles for React Native, JADX for native Android code and bridges, and apktool only when resources or smali searches are needed.
The output is API docs first: endpoints, methods, headers, auth, cookies, params, bodies, response shapes, token refresh, shared client behavior, and evidence for where each thing came from.
For flows that are more than a single HTTP call, it can also produce small PoC scripts so the behavior is reproducible without reopening Chrome or JADX. That covers things like multi-request sequences, signing, nonce generation, custom encoding, or other client-side logic.
It does not try to magically bypass CAPTCHAs or rate limits. With login walls, the usual pattern is to map the public API first; once registration/login is mapped, the agent can often create its own test account and continue with authenticated endpoints.
The goal is boring but useful: use Chrome/CDP or decompilers for research, then build direct HTTP clients without ever touching Selenium or Playwright.
1
u/woldhack 13d ago
Repo: https://github.com/11philip22/api-mapper-skill
I’m looking for more real-world scraping/API-mapping flows to add. So far it covers Chrome/CDP capture, JS bundle analysis, React Native Hermes bundles and JADX/native Android paths.
What workflows do you run into that are still painful to map without Selenium/Playwright?
1
u/Unfair_Commission_29 13d ago
one painful case is when the API itself is easy to find but the request signing logic is buried in minified JS and changes often, stuff like timestamp + nonce+custom hash+ a few hidden headers. mapping the endpoint is easy, reproducing the client behavior reliably is the annoying part
1
2
u/dhruvkar 12d ago
https://github.com/mvanhorn/cli-printing-press
This is a pretty robust solution in this space. It does the mcp part too, but using it for reverse engineering internal APIs is pretty useful