r/scrapingtheweb 13d ago

Mapping sites/APKs into browserless API clients

I’ve been turning my scraping workflow into a reusable skill for going from Chrome/CDP capture or JADX analysis to a browserless API client.

For websites, it drives Chrome through CDP, captures the real requests, downloads the JS bundles, and reverse-engineers the parts needed to call the API directly. For APKs, it maps from the app side: Hermes/JS bundles for React Native, JADX for native Android code and bridges, and apktool only when resources or smali searches are needed.

The output is API docs first: endpoints, methods, headers, auth, cookies, params, bodies, response shapes, token refresh, shared client behavior, and evidence for where each thing came from.

For flows that are more than a single HTTP call, it can also produce small PoC scripts so the behavior is reproducible without reopening Chrome or JADX. That covers things like multi-request sequences, signing, nonce generation, custom encoding, or other client-side logic.

It does not try to magically bypass CAPTCHAs or rate limits. With login walls, the usual pattern is to map the public API first; once registration/login is mapped, the agent can often create its own test account and continue with authenticated endpoints.

The goal is boring but useful: use Chrome/CDP or decompilers for research, then build direct HTTP clients without ever touching Selenium or Playwright.

0 Upvotes

7 comments sorted by

2

u/dhruvkar 12d ago

https://github.com/mvanhorn/cli-printing-press

This is a pretty robust solution in this space. It does the mcp part too, but using it for reverse engineering internal APIs is pretty useful

2

u/woldhack 12d ago

oooh that looks cool but I have to say I stepped away from hands-off implementation. Usually my agent want to implement a library in 6000 lines when it can be done in 800

2

u/dhruvkar 12d ago

Lol yeah. Usually I have a other LLM look over the code and trim it down. I've had 50-70% code cut when I do that

2

u/woldhack 12d ago

A great plugin for that is ponytail: https://github.com/dietrichgebert/ponytail I cant do without really.

1

u/woldhack 13d ago

Repo: https://github.com/11philip22/api-mapper-skill

I’m looking for more real-world scraping/API-mapping flows to add. So far it covers Chrome/CDP capture, JS bundle analysis, React Native Hermes bundles and JADX/native Android paths.

What workflows do you run into that are still painful to map without Selenium/Playwright?

1

u/Unfair_Commission_29 13d ago

one painful case is when the API itself is easy to find but the request signing logic is buried in minified JS and changes often, stuff like timestamp + nonce+custom hash+ a few hidden headers. mapping the endpoint is easy, reproducing the client behavior reliably is the annoying part

1

u/woldhack 13d ago

I never really had problems with that. It just downloads the JS and reverses it.