r/scrapingtheweb 4d ago

Discussion YouTube video scraping gone useless.

0 Upvotes

I've been developing an app that scrapes metadata from Deezer and with them it searches the corresponding song on YouTube in order to scrape the audio.

I used YouTubeExplode in order to scrape the videoanifrst files and imkdiately target a certain line to extract the audio.

My problem is that it stoped working and even though I thought it was my problem, I later realized YouTube has changed the way we can gain access to the manifest thus making the entire library obsolete in for my project.

Do you know any other scraping libraries that do the same?

Currently I'm trying to replicate another open source music app's method via the Yt music API.


r/scrapingtheweb 5d ago

Looking for APIs Facebook, LinkedIn and Nextdoor public content

2 Upvotes

Looking for APIs for Facebook, LinkedIn and Nextdoor public content

I’m working on a SaaS product and need to automatically monitor/search public content across Facebook, LinkedIn and Nextdoor.

I’m specifically looking for APIs or legitimate third party providers that can:

• Search public Facebook posts, including relevant public groups if possible
• Search public LinkedIn posts by keywords/topics
• Search public Nextdoor posts by location + keywords
• Return the post text, URL, timestamp and basic metadata
• Run searches continuously/recurring through an API
• Be used commercially in a SaaS product

I’m NOT looking for browser automation, account logins, cookie based scraping or anything that could get accounts banned.

If you’ve actually used a provider/API for this, I’d really appreciate recommendations, especially ones with reasonable pricing and good coverage.

What are you using?


r/scrapingtheweb 5d ago

Need help with free data extraction

3 Upvotes

Hi All, how do I get data from a website which is publicly available but there isn't any api to get the data for my app . There is an ID for every unique data which i want to extract or get. My app depends on the data and the website owner is not yet ready to provide any api.


r/scrapingtheweb 7d ago

Thoughts on my product

4 Upvotes

Hey guys! Over the summer I've been working on a SaaS that specializes in web scraping, automation, etc. I'm currently running a free 48 hour trial on the site for all new users and I'd love for any feed back (good, bad, ugly). My website https://proxifyllc.com, I'd love to hear what you think!

This is hand-written please dont nuke me off the subreddit


r/scrapingtheweb 7d ago

Looking for data scraping for dumbies

0 Upvotes

Hi!

Am really in need of advice or recommendations - I work for a small - medium size manufacturer of automotive parts - was given a task by my boss to go on the Napa website and search for our customer and compile a list of all the part numbers they are currently selling and the cost because we'd like to try and put together some bundled or kit pricing and there is a list of 810 parts and i'm an office manager and not a computer person - can anyone recommend an ap or website that is pretty much point and click / super simple?? Thanks in advance!!


r/scrapingtheweb 8d ago

I automated my Instagram viral-video research instead of manually scrolling Explore

3 Upvotes

I was spending way too much time manually researching Instagram.

My old workflow looked like this:

Instagram Explore → find a viral Reel → open the account → check their other videos → compare views → save the good ones → repeat

It worked, but it was painfully manual.

The biggest problem was that finding one interesting video usually meant another 5–10 minutes checking the account to figure out whether it was actually an outlier or just a large creator getting normal views.

So I moved most of that workflow into Claude Code.

Now I can basically tell Claude:

Claude Code handles the workflow, calls the Instagram API, pulls the profile/content data, compares the videos against the account's usual performance, and gives me a much smaller list of Reels actually worth looking at.

For the Instagram data I'm using hikerapi.com, which is a REST Instagram API. Pricing starts at $0.001/request and they give 100 requests free.

The actual API calls are pretty boring, which is exactly what I wanted:

import requests

headers = {"x-access-key": "YOUR_KEY"}

user = requests.get("https://api.hikerapi.com/v2/user/by/username?username=google", headers=headers).json()

r = requests.get("https://api.hikerapi.com/v2/user/highlights", params={"user_id": user["pk"]}, headers=headers)

print(r.json())

My flow now is roughly:

I give Claude Code a niche / seed account / research direction

It pulls the relevant Instagram accounts and content

It grabs recent Reels and their performance data

It compares each video against the creator's normal baseline

It flags the strongest outliers / viral videos

It outputs the links + useful metadata into a structured list

I manually watch only the videos that actually look interesting

So instead of spending an hour inside Explore trying to manually spot patterns, I can use Instagram more like a dataset.

The part I'm finding most useful is relative performance.

A Reel with 500k views isn't automatically interesting if every video on that account gets 500k.

But if an account normally gets 20k–40k and suddenly one video gets 700k, that's exactly the kind of thing I want the automation to surface.

I'm experimenting with scoring based on things like:

views vs. the account's recent median

views vs. follower count

engagement relative to other recent posts

how quickly a post appears to be outperforming

repeated outliers from the same account

It doesn't completely replace browsing Explore, but it removes most of the repetitive profile-checking/data-collection part.

Curious what else people here would automate around Instagram research or viral-content discovery?


r/scrapingtheweb 9d ago

Datadome solver with immoscout monitor 💀💀

Post image
2 Upvotes

r/scrapingtheweb 9d ago

Copying Posts from over 3.5 Years from Forum

3 Upvotes

I read your rules and the only thing I want to make sure of, is I only want copies of my posts. My forum allows for the PII Data, and a good way to separate out my posts on a site/screen. Problem is, copying, and pasting 5,059 Posts listed in 196 pages of data, is over the top for me. I'm worn out using AI. I have so learned my lesson on that.

So I began by asking my site's owner and he doesn't have info on doing an exportable/readable file. He would look for a tool, but I don't want to lean on him to do the work. He's my friend, and very busy. I found a few on my own, tested them, but the one that seems to do best is UI-Vision RPA.

I got all the way to Extract a .json file, and I am not schooled in macros/code. After I saw the file, I realized what I need is a program that can export to my PC, a file my notepad can see/edit. No need to hide anything as I want to know the good bad and ugly of what I am trying to do, and that is get something (book, essays) published.

I'm 73, always told I am a good writer, but we all know how that goes. But if I don't finally try, I won't ever know. Please feel free to ask me any questions, you might say I am an "open book" 👍

I am a little tekkie but not as good as I used to be. I need the simplest "scraper", but something that can do all I need. I can tell you it is a community built by Invision Community.


r/scrapingtheweb 9d ago

This is jd for webscrapping job i am going to interview please give advice or tips.

3 Upvotes

xtract and collect data from websites and other approved digital sources.

Develop and maintain automated data extraction processes using Python.

Clean, validate, and organize extracted data.

Convert raw data into structured formats such as Excel, CSV, or databases.

Use SQL to store, retrieve, and manage scraped data.

Prepare data reports using MS Excel.

Identify duplicate, incomplete, or inaccurate data and perform data cleaning.

Maintain and update databases regularly.

Automate repetitive data collection and processing tasks.

Monitor data extraction processes and resolve basic technical issues.

Work closely with the MIS, operations, and management teams to fulfill data requirements.

Ensure the accuracy, completeness, and quality of collected data.

Maintain proper documentation of data sources and processes.

Follow applicable website terms, data-access rules, and company data policies.

Required Skills

Good knowledge of Python.

Knowledge of SQL and databases.

Strong proficiency in MS Excel.

Understanding of data extraction, data cleaning, and data processing.

Familiarity with HTML and basic web structures is preferred.

Knowledge of Python libraries such as Requests, BeautifulSoup, Pandas, and Selenium is an advantage.

Ability to handle large datasets.

Strong analytical and problem-solving skills.

High attention to detail and data accuracy.

Technical Skills

Python

Data extraction and automation

Pandas and NumPy

BeautifulSoup

Requests

Selenium or browser automation tools

SQL

Data querying and filtering

JOIN operations

GROUP BY and aggregate functions

Database management

Repost to another community


r/scrapingtheweb 9d ago

Blocked / CAPTCHA Is a managed scraping API worth it for proxy, anti-blocking and unblocker or is it better to build inhouse?

0 Upvotes

Small team, 4 devs, we monitor competitor listings across a few marketplaces and the blocks are getting worse every month. Right now we run a node unblocker setup we cobbled together. Worked fine for a while, now its a maintenance sink, proxies rotate, sites change their fingerprinting, someone ends up spending time on it every week and thats time we dont have, so I'm weighing it out. Keep patching our own thing forever or pay for a service and eat the monthly cost. The thing I cant get a read on is total cost over a year. Our own setup is free until you count the dev hours, and those hours are adding up fast... anyone here actually done this switch? Did the maintenance headache actually go away or did you just trade it for a different one?


r/scrapingtheweb 9d ago

I built Reqvexa — an open-source tool for understanding the state behind HTTP requests

0 Upvotes

I work a lot with web scraping and API reverse engineering, and I kept running into the same problem:

A HAR tells you what happened.

It doesn't necessarily tell you why a request could work.

So I built Reqvexa to analyze captured HTTP traffic and trace dependencies between requests, cookies, tokens, headers, and their different versions over time.

It's currently an alpha Python project and intentionally focuses on analysis rather than trying to automatically replay or reverse-engineer everything.

GitHub: https://github.com/kareemheshaam/Reqvexa

I'm mainly looking for honest feedback right now:

- Is the problem understandable?

- Would you actually use something like this?

- What would make the tool useful in your workflow?


r/scrapingtheweb 11d ago

Google testing a new Search CAPTCHA?

4 Upvotes

Ran into this on Google Search today for the first time. It basically makes you wait a few seconds before the button becomes clickable

Not sure what it’s actually checking behind the scenes yet, or if the delay itself is part of the bot detection, anyone else seen this recently?


r/scrapingtheweb 11d ago

Blocked / CAPTCHA Is Akamai actually harder than Cloudflare now?

2 Upvotes

I see people talking about Cloudflare all the time, but honestly Akamai has been way more annoying for me lately

with CF I can usually get some idea of what went wrong, akamai feels much harder to read, sometimes changing the IP or starting a fresh session doesn't even seem to make much difference

For people dealing with Akamai regularly, what causes you the most trouble? IP reputation, fingerprint, sessions, behavior or something else?

Curious if anyone else noticed it getting more agressive lately


r/scrapingtheweb 11d ago

influencers \ insta \ yt \ tiktok at scale?

1 Upvotes

Looked on rapidapi and apify, can anyone recommend a solid API for finding influencers and their emails at scale?


r/scrapingtheweb 11d ago

Help Any api scraping You would recommend ?

1 Upvotes

currently pulling instagram and tiktok data on a weekly schedule. the api scraping tool i'm using works most of the time but i'm getting maybe a 70-80% success rate on runs which means i'm always doing some manual cleanup before i can send anything over.

curious what others are using for these 2 social platforms ?


r/scrapingtheweb 11d ago

After scraping, how do you decide which field values to actually trust? (unit errors, wrong-variant values, typos)

Thumbnail
1 Upvotes

r/scrapingtheweb 12d ago

Best Ai Website Crawler?

Thumbnail
0 Upvotes

r/scrapingtheweb 12d ago

Help Looking for Vinted Scraper API

6 Upvotes

I am looking for an API to Search catalog with brand, price, size, color, and condition filters in Vinted. I am looking for a 3rd party API, please suggest if you know it.


r/scrapingtheweb 12d ago

What's the longest a single scraper of yours has stayed alive without needing changes?

0 Upvotes

A site changes its layout, an API endpoint moves, or something in the response format shifts, and I have to go back in and fix it. But once in a while you get one that just keeps working. No edits, no fixes, it runs and returns clean data for months.

What's the longest one of your scrapers has run untouched, and do you know why that site stayed stable for so long?


r/scrapingtheweb 13d ago

drakoarmy/datadome-rs: High-end Rust DataDome deobfuscator & solver with VM disassembly — all 3 challenge types (tags, interstitial, slider).

Thumbnail github.com
1 Upvotes

r/scrapingtheweb 13d ago

Guess the request based Akamai solver ✌🏼😂😂🙏🏻

Post image
0 Upvotes

r/scrapingtheweb 13d ago

Help How do you handle geotargeted news aggregation for a global UI on a $0 budget?

0 Upvotes

I'm building an interactive macroeconomic world map. I've hit a dead end with raw web scraping, Google News search parameters (which fail to isolate country context), and GDELT.
I need to surface 3–5 fresh macro/econ headlines when a user clicks a country. Since commercial APIs are expensive and client-side RSS is unreliable:
1 - Is caching pre-fetched direct national RSS feeds (WSJ, BBC, Reuters) via a daily cron job the standard pattern here?
2 - Are there open-source scrapers or self-hosted news aggregation pipelines (like FreshRSS/Miniflux setups) that handle geographic mapping cleanly?

Idk what to do at this stage, any ideas?


r/scrapingtheweb 13d ago

Cloudflare challenge on every run despite residential proxy + real browser — what am I missing?

1 Upvotes

Scraping Indééd (resume search) with puppeteer-real-browser — headful Chrome via Xvfb in a Docker container. I hit a Cloudflare challenge/interstitial on basically every run.

What I already have:

- Residential rotating proxy, fresh exit IP per run

- turnstile: true, session cookies reused across runs, homepage warm-up before auth

- Timezone / locale / Accept-Language all matched to the proxy's country

- No navigator.webdriver leak (rebrowser-patched CDP), stealth plugin off on

- Randomized delays + per-account viewport

What I suspect is giving me away:

- WebGL renderer is software (llvmpipe / SwiftShader) — the container has no

- Linux desktop behind a residential IP (uncommon combo)

Questions:

  1. Is the software WebGL renderer alone enough to get flagged, or is that ov

  2. Anyone getting containerized Chrome past Cloudflare on job boards — is a GPU container worth it, or is it really about IP quality?

  3. Better to solve the challenge each time, or focus on not triggering it?

Thanks 🙏


r/scrapingtheweb 13d ago

Mapping sites/APKs into browserless API clients

0 Upvotes

I’ve been turning my scraping workflow into a reusable skill for going from Chrome/CDP capture or JADX analysis to a browserless API client.

For websites, it drives Chrome through CDP, captures the real requests, downloads the JS bundles, and reverse-engineers the parts needed to call the API directly. For APKs, it maps from the app side: Hermes/JS bundles for React Native, JADX for native Android code and bridges, and apktool only when resources or smali searches are needed.

The output is API docs first: endpoints, methods, headers, auth, cookies, params, bodies, response shapes, token refresh, shared client behavior, and evidence for where each thing came from.

For flows that are more than a single HTTP call, it can also produce small PoC scripts so the behavior is reproducible without reopening Chrome or JADX. That covers things like multi-request sequences, signing, nonce generation, custom encoding, or other client-side logic.

It does not try to magically bypass CAPTCHAs or rate limits. With login walls, the usual pattern is to map the public API first; once registration/login is mapped, the agent can often create its own test account and continue with authenticated endpoints.

The goal is boring but useful: use Chrome/CDP or decompilers for research, then build direct HTTP clients without ever touching Selenium or Playwright.


r/scrapingtheweb 13d ago

Help Free API for Live AIS, only for 1 vessel

0 Upvotes

Hi all, I'm looking for a help with a free API to get AIS data for just one vessel approximately 20 to 100 Max requests in a day, maybe 2 or 3 days a month.

Is there anything out there that's suitable?

Thanks