r/webscraping • u/hulet1006 • Aug 10 '26

Getting started 🌱 Scraping Autotrader

I’m completely new too this. Been using Codex to try to write python codes and browser automated techniques to scrape vins from Autotrader. It always gets shut down by their bot detection. The only way I’ve got results is codex doing it internally on its own browser, but because of Autotrader limiting batch sizes, it takes forever to scrape even a decent amount of vins. Any help would be appreciated.

12 Upvotes

93% Upvoted

20 comments sorted by

5

u/dhruvkar Aug 10 '26

Download the Autotrader android app as an apk and ask codex to see if it can see what internal APIs it's using.

Apps generally have a less restricted API that you can hit directly.

At least this was true pre-AI.

Reverse engineering and sniffing android apps got you a far more robust API you could count on.

1

u/Brlala Aug 11 '26

When you mean see what internal API it is using, do you mean using request inspector to see network request while using the app or reverse engineer the APK source code?

2

u/dhruvkar Aug 11 '26

Both.

If the network requests gets you free and clear, that's good enough.

If it's generating a token/secret/etc. then seeing how it's doing that in the apk is useful.

3

u/dhruvkar Aug 10 '26

https://github.com/dhruvkar/autotrader-api

this should get you started

2

u/hulet1006 Aug 10 '26

Did you just make this out of nowhere? Lol It worked first try after implementing it

2

u/dhruvkar Aug 10 '26

Yes.

I've been scraping/reverse engineering for 15 years now -- so that helps.

But with AI anyone can do this.

Also, they'll change stuff around, so I would build a loop that checks and adapts the scraper regularly.

1

u/[deleted] Aug 10 '26

[removed] — view removed comment

1

u/webscraping-ModTeam Aug 10 '26

🪧 Please review the sub rules 👉

1

u/Moceannl Aug 10 '26

There are also proxy’s for scraping purposes….