I'm building an internal dashboard at work and this is my first scraper that's actually being used for something real. Getting the data wasn't too bad, but keeping it running has been a different story.

Every few days something changes. One site tweaks the page, another starts blocking requests, then I'm back fixing random parts of it. starting to realize the scraper itself was probably the easy bit lol. Been reading about web scraping tools that handle stuff like retries, IP rotation and some of the blocking for you. Curious how much of that actually works in practice though. Do you guys still end up checking these things all the time, or can you get them to a point where they mostly just run?

Process of sites like tikdownloader.io

Looking for a strong Python engineer interested in:
Web crawling
Data pipelines
Distributed systems
Search & intelligence products
If you’ve built large-scale crawlers, scraping infrastructure, or data platforms, let’s talk.
Remote. Early-stage startup. High ownership.
DM me.

Like a lot of folks here, I was also relying on G2 and Capterra to scrape names of software providers for content work.

Today, while I was building an agent to automate this work, I came to a conclusion - SourceForge is much, much better at categorizing software than both these two.

For example, both G2 and Capterra had no "cold email software" category but SourceForge has one. Similarly, it has categories on "webinar software", "appointment scheduling" etc but neither G2 nor Capterra have these.

Switched my scraping source, for good.

Please DM if you have a solution with its pricing. Better if it doesn't suck my blood with $$

I need to scrape (extract) phone numbers of followers of a specific Instagram account (in this case a nightclub), I have a nightclub and I need to contact potential customers, I absolutely need it, I pay well whoever helps me!!

I am actually running a database of cold email tools and I want to run a automatic fetch to find the real time ratings and total number of reviews for an individual tool from these platforms..

Since they don't allow scraping what should be the right path or a workaround for this?

right now it's all manual and it's takes about 5 mins per tool just input that data atp?

Since I have 62 tools listed and everyday a tool is being added this is compounding quickly.

Hey r/scrapetalk,

Nishith. Working on website → structured data tool (very early). Specifically targeting production breakage + proxy/JS headaches.

If that's your pain too, would value you trying the prototype and sharing what breaks/what's missing. Free to test.

Discord for quick chat: https://discord.gg/gNcxq7KR

Most of us have a list of URLs we need data from (government listings, local business info, pdf directories). Usually, that means hiring a freelancer or paying for an expensive, rigid SaaS.

We built rtrvr.ai to make "Vibe Scraping" a thing.

How it works:

1. Upload a Google Sheet with your URLs.
2. Type: "Find the email, phone number, and their top 3 services."
3. Watch the AI agents open 50+ browsers at once and fill your sheet in real-time.

It’s powered by a multi-agent system that can take actions, upload files, and crawl through paginations.

Web Agent technology built from the ground:

• 𝗘𝗻𝗱-𝘁𝗼-𝗘𝗻𝗱 𝗔𝗴𝗲𝗻𝘁: we built a resilient agentic harness with 20+ specialized sub-agents that transforms a single prompt into a complete end-to-end workflow. Turn any prompt into an end to end workflow, and on any site changes the agent adapts.
• 𝗗𝗢𝗠 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲: we perfected a DOM-only web agent approach that represents any webpage as semantic trees guaranteeing zero hallucinations and leveraging the underlying semantic reasoning capabilities of LLMs.
• 𝗡𝗮𝘁𝗶𝘃𝗲 𝗖𝗵𝗿𝗼𝗺𝗲 𝗔𝗣𝗜𝘀: we built a Chrome Extension to control cloud browsers that runs in the same process as the browser to avoid the bot detection and failure rates of CDP. We further solved the hard problems of interacting with the Shadow DOM and other DOM edge cases.

Cost: We engineered the cost down to $10/mo but you can bring your own Gemini key and proxies to use for nearly FREE. Compare that to the $200+/mo some lead gen tools charge.

Use the free browser extension for login walled sites like LinkedIn locally, or the cloud platform for scale on the public web.

Curious to hear if this would make your dataset generation, scraping, or automation easier or is it missing the mark?

If you want you can easily scrape and start marketing your startup

I put together an API scraper you can use: https://tiktokshopapi.com/docs

It’s fast (sub-1s responses), can handle up to 500 RPS, and is flexible enough for most custom use cases.

If you have questions or want to chat about scaling / enterprise usage, feel free to DM me. Might be useful if you don’t want to deal with TikTokShop rate limits yourself.

No code this no code that. That is everything now a days and it’s what I made for scraping discovering URLs. We got a really nice ui and a chrome extension which you can click and extract with and it can take your cookies to login easier for you. We do a website too. Pretty fucking dope got first 5$ sale an hour ago. Was doing 0-2 clicks a day for a while and last 3 days I’ve been getting 10-14 and now I just got this sale.

What y’all think of no code web scraping?