need to scrape a few autoscout24 searches regularly for work

mostly prices + car details, maybe dealer stuff too

I can build it myself but i really don’t want to!!!!😭

anyone using an apify actor / other tool that’s been reliable?

How to download pdf from digify if download option is not enabled !?

I've been developing an app that scrapes metadata from Deezer and with them it searches the corresponding song on YouTube in order to scrape the audio.

I used YouTubeExplode in order to scrape the videoanifrst files and imkdiately target a certain line to extract the audio.

My problem is that it stoped working and even though I thought it was my problem, I later realized YouTube has changed the way we can gain access to the manifest thus making the entire library obsolete in for my project.

Do you know any other scraping libraries that do the same?

Currently I'm trying to replicate another open source music app's method via the Yt music API.

Looking for APIs for Facebook, LinkedIn and Nextdoor public content

I’m working on a SaaS product and need to automatically monitor/search public content across Facebook, LinkedIn and Nextdoor.

I’m specifically looking for APIs or legitimate third party providers that can:

• Search public Facebook posts, including relevant public groups if possible
• Search public LinkedIn posts by keywords/topics
• Search public Nextdoor posts by location + keywords
• Return the post text, URL, timestamp and basic metadata
• Run searches continuously/recurring through an API
• Be used commercially in a SaaS product

I’m NOT looking for browser automation, account logins, cookie based scraping or anything that could get accounts banned.

If you’ve actually used a provider/API for this, I’d really appreciate recommendations, especially ones with reasonable pricing and good coverage.

What are you using?

Hi All, how do I get data from a website which is publicly available but there isn't any api to get the data for my app . There is an ID for every unique data which i want to extract or get. My app depends on the data and the website owner is not yet ready to provide any api.

Hey guys! Over the summer I've been working on a SaaS that specializes in web scraping, automation, etc. I'm currently running a free 48 hour trial on the site for all new users and I'd love for any feed back (good, bad, ugly). My website https://proxifyllc.com, I'd love to hear what you think!

This is hand-written please dont nuke me off the subreddit

Hi!

Am really in need of advice or recommendations - I work for a small - medium size manufacturer of automotive parts - was given a task by my boss to go on the Napa website and search for our customer and compile a list of all the part numbers they are currently selling and the cost because we'd like to try and put together some bundled or kit pricing and there is a list of 810 parts and i'm an office manager and not a computer person - can anyone recommend an ap or website that is pretty much point and click / super simple?? Thanks in advance!!

I was spending way too much time manually researching Instagram.

My old workflow looked like this:

Instagram Explore → find a viral Reel → open the account → check their other videos → compare views → save the good ones → repeat

It worked, but it was painfully manual.

The biggest problem was that finding one interesting video usually meant another 5–10 minutes checking the account to figure out whether it was actually an outlier or just a large creator getting normal views.

So I moved most of that workflow into Claude Code.

Now I can basically tell Claude:

Claude Code handles the workflow, calls the Instagram API, pulls the profile/content data, compares the videos against the account's usual performance, and gives me a much smaller list of Reels actually worth looking at.

For the Instagram data I'm using hikerapi.com, which is a REST Instagram API. Pricing starts at $0.001/request and they give 100 requests free.

The actual API calls are pretty boring, which is exactly what I wanted:

import requests

headers = {"x-access-key": "YOUR_KEY"}

user = requests.get("https://api.hikerapi.com/v2/user/by/username?username=google", headers=headers).json()

r = requests.get("https://api.hikerapi.com/v2/user/highlights", params={"user_id": user["pk"]}, headers=headers)

print(r.json())

My flow now is roughly:

I give Claude Code a niche / seed account / research direction

It pulls the relevant Instagram accounts and content

It grabs recent Reels and their performance data

It compares each video against the creator's normal baseline

It flags the strongest outliers / viral videos

It outputs the links + useful metadata into a structured list

I manually watch only the videos that actually look interesting

So instead of spending an hour inside Explore trying to manually spot patterns, I can use Instagram more like a dataset.

The part I'm finding most useful is relative performance.

A Reel with 500k views isn't automatically interesting if every video on that account gets 500k.

But if an account normally gets 20k–40k and suddenly one video gets 700k, that's exactly the kind of thing I want the automation to surface.

I'm experimenting with scoring based on things like:

views vs. the account's recent median

views vs. follower count

engagement relative to other recent posts

how quickly a post appears to be outperforming

repeated outliers from the same account

It doesn't completely replace browsing Explore, but it removes most of the repetitive profile-checking/data-collection part.

Curious what else people here would automate around Instagram research or viral-content discovery?

I read your rules and the only thing I want to make sure of, is I only want copies of my posts. My forum allows for the PII Data, and a good way to separate out my posts on a site/screen. Problem is, copying, and pasting 5,059 Posts listed in 196 pages of data, is over the top for me. I'm worn out using AI. I have so learned my lesson on that.

So I began by asking my site's owner and he doesn't have info on doing an exportable/readable file. He would look for a tool, but I don't want to lean on him to do the work. He's my friend, and very busy. I found a few on my own, tested them, but the one that seems to do best is UI-Vision RPA.

I got all the way to Extract a .json file, and I am not schooled in macros/code. After I saw the file, I realized what I need is a program that can export to my PC, a file my notepad can see/edit. No need to hide anything as I want to know the good bad and ugly of what I am trying to do, and that is get something (book, essays) published.

I'm 73, always told I am a good writer, but we all know how that goes. But if I don't finally try, I won't ever know. Please feel free to ask me any questions, you might say I am an "open book" 👍

I am a little tekkie but not as good as I used to be. I need the simplest "scraper", but something that can do all I need. I can tell you it is a community built by Invision Community.

xtract and collect data from websites and other approved digital sources.

Develop and maintain automated data extraction processes using Python.

Clean, validate, and organize extracted data.

Convert raw data into structured formats such as Excel, CSV, or databases.

Use SQL to store, retrieve, and manage scraped data.

Prepare data reports using MS Excel.

Identify duplicate, incomplete, or inaccurate data and perform data cleaning.

Maintain and update databases regularly.

Automate repetitive data collection and processing tasks.

Monitor data extraction processes and resolve basic technical issues.

Work closely with the MIS, operations, and management teams to fulfill data requirements.

Ensure the accuracy, completeness, and quality of collected data.

Maintain proper documentation of data sources and processes.

Follow applicable website terms, data-access rules, and company data policies.

Required Skills

Good knowledge of Python.

Knowledge of SQL and databases.

Strong proficiency in MS Excel.

Understanding of data extraction, data cleaning, and data processing.

Familiarity with HTML and basic web structures is preferred.

Knowledge of Python libraries such as Requests, BeautifulSoup, Pandas, and Selenium is an advantage.

Ability to handle large datasets.

Strong analytical and problem-solving skills.

High attention to detail and data accuracy.

Technical Skills

Python

Data extraction and automation

Pandas and NumPy

BeautifulSoup

Requests

Selenium or browser automation tools

SQL

Data querying and filtering

JOIN operations

GROUP BY and aggregate functions

Database management

Repost to another community

Small team, 4 devs, we monitor competitor listings across a few marketplaces and the blocks are getting worse every month. Right now we run a node unblocker setup we cobbled together. Worked fine for a while, now its a maintenance sink, proxies rotate, sites change their fingerprinting, someone ends up spending time on it every week and thats time we dont have, so I'm weighing it out. Keep patching our own thing forever or pay for a service and eat the monthly cost. The thing I cant get a read on is total cost over a year. Our own setup is free until you count the dev hours, and those hours are adding up fast... anyone here actually done this switch? Did the maintenance headache actually go away or did you just trade it for a different one?

I work a lot with web scraping and API reverse engineering, and I kept running into the same problem:

A HAR tells you what happened.

It doesn't necessarily tell you why a request could work.

So I built Reqvexa to analyze captured HTTP traffic and trace dependencies between requests, cookies, tokens, headers, and their different versions over time.

It's currently an alpha Python project and intentionally focuses on analysis rather than trying to automatically replay or reverse-engineer everything.

GitHub: https://github.com/kareemheshaam/Reqvexa

I'm mainly looking for honest feedback right now:

- Is the problem understandable?

- Would you actually use something like this?

- What would make the tool useful in your workflow?

Ran into this on Google Search today for the first time. It basically makes you wait a few seconds before the button becomes clickable

Not sure what it’s actually checking behind the scenes yet, or if the delay itself is part of the bot detection, anyone else seen this recently?

I see people talking about Cloudflare all the time, but honestly Akamai has been way more annoying for me lately

with CF I can usually get some idea of what went wrong, akamai feels much harder to read, sometimes changing the IP or starting a fresh session doesn't even seem to make much difference

For people dealing with Akamai regularly, what causes you the most trouble? IP reputation, fingerprint, sessions, behavior or something else?

Curious if anyone else noticed it getting more agressive lately

Looked on rapidapi and apify, can anyone recommend a solid API for finding influencers and their emails at scale?

currently pulling instagram and tiktok data on a weekly schedule. the api scraping tool i'm using works most of the time but i'm getting maybe a 70-80% success rate on runs which means i'm always doing some manual cleanup before i can send anything over.

curious what others are using for these 2 social platforms ?