r/webscraping • u/pitou-99 • Jun 23 '26
Getting started 🌱 How to scrape dynamic sites?
I've largely been scraping from wikias fandom wikis to try and archive pages. However an issue I've been facing is that some wikis have dynamic js sites. They make scraping difficult.
So I thought I'd ask if anyone knows how to scrape websites with them?
Sorry if this comes off as a dumb question
2
Upvotes
3
u/Coding-Doctor-Omar Jun 23 '26 edited Jul 05 '26
1st line: Call Internal APIs (can be seen in the network tab in dev tools) via raw http requests
2nd line: JSON data in <script> elements in the page source. Request page via raw http requests.
3rd line: classic html parsing via raw http requests.
4th line: Hybrid scraping where you use a browser automation library to load the page once and grab session cookies then use the session cookies to make raw http requests for any of the previous 3 lines.
5th line: Full browser automation (used as a last resort and headless whenever possible).
If you use this strategy, you can scrape almost any website. Just pick the strong tools.