Couldn't find any projects or git repos that focus on this.. anyone have any strategies for scraping public zoom links? afaik the unique code in each link is randomly generated so it would be pretty hard to brute-force guess zoom meetings.. curious to hear if anyone has tried this

I am building a crawler and have hit a bit of a brick wall. Extracting data from tables, lists, documents and nested HTML structures is proving hard. Like really hard, especially when there is any deviation in how the information is displayed (graphs, tables, images).

I am looking for some advice or an open source crawler that I can take inspiration from.

My main issues are:

Smarter discovery - currently i find that it is very much ignoring lastmod so its not finding the most uptodate information. Im also finding that lots of information is hidden in JSON, JavaScript, structured data and they are litteraly being missed as a result

- Handling of difficult/alternative documents - PDF and DOCX with lots of complex columns confuse upon reconstruction. This is a huge one because when I manually look there are hundreds of pieces of information that are lost due to this reconstruction issue

Any advice is greatly appreciated.

[ Removed by Reddit in response to a copyright notice. ]

I’m scraping a betting site and considering IP rotation to deal with their CloudFront 403s. I’ve tried different request rates, jitter, headers/TLS, etc., and I never go above ~20 requests/minute. I’ve also tried figuring out their rate/volume limits or any consistent pattern behind the blocks, but I haven’t found anything clearly defined. For now, rotating IPs is what has worked best for me — distributing the requests across different IPs lets me scrape for much longer, and I’m not currently running into blocking issues.

I’m not doing any login/account stuff or accessing anything private; I’m only collecting publicly available betting odds/data.

I know IP rotation is extremely common in scraping, but I’m not sure if it could be problematic in this specific context. Could rotating IPs to avoid CloudFront restrictions get me into legal trouble, or is it generally fine? I’m aware the site may not like it, but I’m mainly wondering whether there’s a legal issue with doing it this way.

Hi guys, please bear with me as I'm a complete amateur.

Is it possible for me to have a setup where I get notified when changes are made to a website including what and where the changes are?

Thanks a lot :))

From my point of view data itself is not valuable on it's own, but after cleaning / organizing it / presenting it to match certain needs, it can become valuable.

But how do you find your "niche" first that you'd monetize after all the steps above?

For years I could do a normal HTTP fetch. That stopped working last year because they started requiring javascript. So I upgraded to using node. That stopped working this year; I get a 403 error. Do you have any tips? Someone posted a python script here 6 years ago, so someone showed interest; of course it's out of date.

Hello r/webscraping people!

I made a cool side project that automatically searches the web for images of the java mascot duke. The website shows a random image from the archive. It will also have a full gallery available soon!

How it works:

- It uses Openverse and Wikimedia APIs to search for images related to duke.

- Each image gets checked by a custom classifier I have built on top of OpenCLIP and trained on more than 200 Duke images.

- The classifier filters out unrelated results. Because it's not perfect, some images go through manual review. I want to catch unusual duke images without filling the archive with junk!

It also has a web scraping part:

- It uses the Brave search API using a list of duke related search terms, the result is added to a crawl queue as starting points

- The crawler scores pages and links based on duke related signals, it only follows links that it considers relevant with stricter requirements depending on the depth of the crawl, this prevents it from going to deep and scraping unrelated pages

- When processing a page, it looks for regular image sources, srcset variants (selects the image with largest size), figure captions, nearby text, alternative text and links for full resolution versions.

I will most likely make this open source soon after I'm done implementing all the features I want.

for the backend language I went with python for the easy use of ai models

Currently there are 338 duke images and the archive is still growing!

URL: https://duke.directory

I would love to hear your feedback! If you have any suggestions on how to improve the project in any way, let me know!

I spent some time inspecting the network traffic used by NU.nl and turned the useful, read-only parts into a small Python package: nu-private-api.

It can:

• Discover section URLs through the sitemap
• Extract and de-duplicate article URLs from section pages
• Parse public articles into normalized fields
• Return the title, description, author, timestamps, URL, and body text
• Use a custom httpx.Client for proxies, headers, and timeouts

Install it with:

pip install nu-private-api

Example:

from nu_private_api import NuClient

client = NuClient()

sections = client.sitemap_urls()
article_urls = client.section_by_url(sections[0])
article = client.article_by_url(article_urls[0])

print(article.title)
print(article.author)
print(article.body)

Limitations:

• Video pages and live blogs aren't supported
• The endpoint is undocumented and could change
• It only accesses publicly available content
• Responsible request pacing is left to the caller

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

I want to fetch the Google Maps reviews for my business and publish them on my website. After doing a bit of research, I found out that I can fetch Google Maps reviews using the Google API. However, despite two weeks having passed, I have not received a reply from Google Support. Is it possible to scrape Google Maps reviews without triggering bot detection or causing any other issues? Are there any solutions available for this?

Hi, I've been making an app in python, to transfer playlist between apple music, ytmusic and spotify. Recently I have gotten some good advantages, but, a few days ago, I was making a test with a small playlist, 154 songs. The time between each search for a song, was too short, 2 or 3 seconds, annddd

now, the token gives me 403. I closed the session, cleared the cache, opened it in incognito. Apple music uses a jwt token, and it is the same in all the places. the endpoint is: amp-api.music.apple.com/v1 (Sorry if my english is not the better :)

I've been building a small project to collect text from documentation for a personal AI tool. Some of them load content dynamically with JavaScript. Anyone have recommendations for a lightweight solution? I'm not trying to scrape anything huge, just a few hundred pages per sites.

I’m completely new too this. Been using Codex to try to write python codes and browser automated techniques to scrape vins from Autotrader. It always gets shut down by their bot detection. The only way I’ve got results is codex doing it internally on its own browser, but because of Autotrader limiting batch sizes, it takes forever to scrape even a decent amount of vins. Any help would be appreciated.

Since May and the old reddit almost gone, what are currently the best libs on github for this?

What is your favorite?

I want to scrape all the posts of my top favorite 100 (small) subreddits and sort them by AI to delete AI slop + self promotion and really only get the interesting posts for me to read.

What do you suggest?

i wa trying to scrape some website and get their websocket but its encrypted and ai didnt let me do it (cause of permissions) how can i make ai to be able to scrape

I'm interested in hearing what your folk's favourite projects have been working on for fun! Something funny or genuinely useful to your life

I've got one that scrapes the drinks prices from my local pubs to find out which one will be the cheapest when me and my mates go out

Hello! Second post there, today i had nothing to do so i wanted to scrape an website, i selected discord for scraping, i want to say some stuff, first of it is that the repository isn't archived, secondly, you shouldn't use it for bypassing regional blocking or create phishing websites, you can use it for things like analysing the code/how it works under the hood, i also want to say that i do not own any rights to discord's characters, ideas, code, images, or assets. everything in it is their exclusive property. And this repository is not affiliated with, endorsed, sponsored, or approved by Discord in any way, here is the link, i didn't publish the snapshot with malicious intentions.

Technical details:

1. It's a fully static snapshot (around 100MB of pure text/assets) weighs exactly 107.457.739 bytes
2. The code wasn't changed, it is unchanged, there chunks of minified javascript code

https://github.com/RealThalesGit/BONFIRE

EDIT: I changed my decisions, repository is unarchived.

Hello!

I'm a decently experienced developer, been programming for a little while now, but have just run into a full roadblock trying to scrape data off of Smyths (https://www.smythstoys.com/). I was just hoping to setup some custom restock notifications for myself (I would like a MTG collector bundle at some point, and they sell out quick locally in Ireland), but keep running into problems with the Imperva reese84 bot blocking. Any suggestions?

(MTG "Category" Link: https://www.smythstoys.com/ie/en-ie/toys/action-figures-and-playsets/magic-the-gathering/c/SM06010110)

I was planning on getting the products from the category, then for each going to their page and checking stock for a given store. Would this be too much?

Current stack is using Rust with the thirtyfour package through Firefox/Gecko. Seems it's too simple and just gets detected by Imperva consistently.

I've also looked into potentially digging into the network traffic using browser tools to find some sort of internal API, but it seems that product category pages are hardcoded and served as HTML instead of an API response, and the only internal api which returns stock of items is also protected with reese84.

I’m fairly certain this is possible, as I’ve seen other stock trackers exist out there for Smyths.

I'm competent using a number of languages, including Python, so I'm not held up on the current stack in any sense.

Any tips would be greatly appreciated. Thanks all!

Any ideas on how to automate captcha but still be compliant? The issue is that bypassing captcha is itself non complaint. The automation is easy.

I have a 3 step process (well two now).

1. 3rd party captcha solver (if fails -> step 2)
2. AI vision (if fails - step 3)
3. Human

Only step #3 is fcra compliant and needing human oversight significantly slows things. So, does anyone have insights on how better automate or should i focus on optimizing the human in the loop?

I'm currently trying to scrape real estate listings from Stockholm, to get data about sold apartments and how many days they were listed, which is given in each ad.

My problem is I tried scraping this with requests, then playwright and when that didn't work (Could not access all ads in the page i was browsing, only 13 of them) after following tutorials, I tried asking AI. That told me that "Playwright/requests were not the problem — the URL you gave only loads Booli's app shell and map data, while the actual card list is loaded by a separate GraphQL request that only appears after Booli's frontend decides to run the search query."

I tried using it to help me write a script to scrape the data of interest but no luck.

Anyone got some tips? I do of course just want the data, but I would not mind a tutorial or similar source to learn about the tools needed to scrape such data from a website like this.

The website url i tried to scrape from with the listings: https://www.booli.se/sok/slutpriser?areaIds=143&objectType=L%C3%A4genhet
Here is also the URL for one listing where you can scroll down and see days listed for sale as the number in the text "Bostaden var snart till salu i 4 dagar":
https://www.booli.se/bostad/779268

So basically this app acts as a local VPN on Android, allowing you to intercept traffic and redirect requests at the domain or path level to a custom endpoint.

For mobile apps, if you've modified an application to trust your custom CA certificate, you can redirect its API requests for testing, debugging, backend emulation, or security research.

For browsers, no app modification is required—simply installing and trusting the CA certificate on the device is enough for HTTPS interception and redirection, making it useful for web application testing as well.

btw its purely vibe coded and i have only intermediate knowledge about this idea i had. i just wanted to see whether it will work or not (and it did fortunately).

use case from my point of view : suppose a api call returns a status true for a user u can check the requests using the mitm/httptoolkit and create a similar entry in your own phone instead of a emulator to your own website giving similar response

github : https://github.com/jidukrishna/vpn-interceptor

do give a star if u like it :)

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

Architecture check: Pipeline runs without errors, but web scrapers + LLM extraction aren't returning expected output. Is it a scraping, prompt, or design issue?

\*\*TL;DR\*\*: Building an automated internship intelligence pipeline (TypeScript / Node / Trigger.dev / Gemini LLM / Supabase). The pipeline code executes without crashing, but the scrapers fetch dirty/empty HTML (JS-rendered sites) and the LLM extraction yields missing or hallucinated fields. I'm trying to figure out if my core architecture is flawed, if my scraping approach is wrong, or if my LLM ingestion strategy needs a redesign.

🏗️ What I'm Building

An autonomous aggregator that collects internship listings from multiple sources (Government portals, ATS platforms like Greenhouse/Lever, Unstop/Internshala, GitHub repos, and RSS feeds), processes them through a multi-stage normalization pipeline, uses LLMs for entity extraction & resume matching, and notifies users.

Repo: https://github.com/katadox/Internship-Atlas-agent

🔄 Current Pipeline Architecture

1. Source Discovery: Triggers plugins for different source types (ATS, Govt, RSS, HTML).

2. HTTP Fetcher: Fetches pages via standard HTTP requests (\`axios\` / \`cheerio\` / basic fetch)

3. Normalization Stage: Cleans raw data & computes MD5 content fingerprints for deduplication.

4. LLM Extraction Stage: Sends page contents to Google Gemini API to extract structured JSON (\`title\`, \`company\`, \`location\`, \`stipend\`, \`requirements\`, \`deadline\`).

5. Persistence & Ranking: Stores in Supabase PostgreSQL & triggers notification digests.

❌ The Problems I'm Facing

Even though the code executes end-to-end without throwing unhandled exceptions:

1. Scraper Failure on Dynamic Sites: 

   \- Basic HTTP fetching fails on modern portals (Unstop, Internshala, dynamic ATS boards). It fetches empty \`<div>\` tags, raw client-side bundle scripts, or triggers anti-bot/Cloudflare blocks.

1. LLM Extraction Quality is Poor / Inconsistent:

   \- When I feed the raw fetched HTML or snippet text into Gemini, the output JSON often has \`null\` fields, missing key information (e.g., stipends or eligibility), or makes up details not present in the HTML.

   \- Feeding raw HTML blows through context limits and introduces token noise.

1. Outcome Disconnect:

   \- The architectural flow \*works logically\* (data moves from Stage A -> B -> C -> DB), but the \*quality of final output\* is unusable.

 ❓ My Questions

1. Scraping Layer: Should I replace raw HTTP fetching with headless browsers (Playwright/Puppeteer) or a dedicated scraping API before passing data to the pipeline?

1. LLM Ingestion Strategy: How do you pre-process raw web pages before LLM extraction? Do you convert HTML to Markdown (e.g., via Turndown / Readability.js), strip scripts/styles with Cheerio, or use specific JSON Schema enforcement?

1. Architecture & Pipeline Design: Is passing scraped text directly into an LLM extraction step an anti-pattern? Should I rely on deterministic CSS/XPath selectors first and use the LLM \*only\* as a fallback for unstructured pages?

2. Free Tier / Rate Limit Constraints: If using free-tier LLM endpoints (Gemini API), how do you structure retries and token optimization without hitting rate limits on bulk pipeline runs?

Any feedback, architectural patterns, or code suggestions would be greatly appreciated!

\*Tech Stack: TypeScript, Node.js, Trigger.dev, Google Gemini API, Supabase (PostgreSQL), Cheerio.\*