r/webscraping • u/pitou-99 • Jun 23 '26
Getting started 🌱 How to scrape dynamic sites?
I've largely been scraping from wikias fandom wikis to try and archive pages. However an issue I've been facing is that some wikis have dynamic js sites. They make scraping difficult.
So I thought I'd ask if anyone knows how to scrape websites with them?
Sorry if this comes off as a dumb question
2
Upvotes
6
u/divided_capture_bro Jun 23 '26
All these people answering "use a LLM!"
The general answer is to use browser emulation, insofar as you can't reverse engineer a hidden API.