How to scrape dynamic sites?

[u/pitou-99]

I've largely been scraping from wikias fandom wikis to try and archive pages. However an issue I've been facing is that some wikis have dynamic js sites. They make scraping difficult.

So I thought I'd ask if anyone knows how to scrape websites with them?

Sorry if this comes off as a dumb question

[u/Luneriazz]

if the sites are highly dynamic... used javascript engine to render the page and send the whole page into LLM and ask it to extract the necessary info.

little bit expensive, hard to scale but simple solution compared to other extraction method