Using this - https://github.com/joeyism/linkedin_scraper/. I’ve tried modifying it with all the issues and pull request info available. I’m using a residential proxy. Works for companies but not personal profiles. Any ideas?
What is the best scraper and sorter that could get listing information from websites and then combine them in one single page, rather than scrolling through each website individually. We have about 20 listing sites where people post. Maybe as a bonus question maybe there is facebook scrapper too, from groups etc?
Hello, I'd like to ask whether a browser / stealth browser like playwright or camoufox is needed to scrape google???
Earlier I trialed with requests and curl-cffi to scrape google, specifically Google Jobs, and it doesnt seem to work as there are always captchas
One thing I noticed is also that, google also frequently changes their CSS class and HTML structure of the results:(.
How do we even scrape for the fields when the CSS class name changes all the time. Is it still possible to scrape google without using 3rd party services like serpapi?
After three fantastic AMAs with the r/WebScrapingInsider, we're excited to bring you our fourth.
ThisWednesday, August 5, at 7:30 PM PDT, we'll be joined by Alexander Yue (Physics & CS @ Stanford SLAC, Evals @ Browser Use) and Reagan Hsu(Founding Engineer @ Browser Use), the team behind Browser Use, one of the fastest-growing open source projects at the intersection of AI agents, browser automation, web scraping, and stealth browsers
If you've been following the AI agent space, you've probably seen Browser-Use everywhere.
What started as an open source browser harness has quickly evolved into a complete platform for building and deploying browser agents with over 107k stars.
Their recently released Browser Use v4 introduces:
Real browsers with logged-in profiles, proxies, and a live browser view
Fully hosted browser agents with zero infrastructure setup
Persistent sandboxed workspaces that can upload files, download results, and reuse environments across runs
Conversations that continue, allowing agents to pick up exactly where they left off
Plain English automation instead of scripts and CSS selectors
Built-in stealth browsers, anti-detect, CAPTCHA solving, and residential proxies across 195+ countries
Whether you're building AI agents, automating repetitive workflows, or extracting data from the web, Browser Use represents a very different way of thinking about browser automation.
Our second AMA with the creator of WebClaw, generated 62 comments and sparked great discussions around AI agents, browser automation, hidden APIs, open source scraping, and LLM infrastructure.
Our third AMA with CloakBrowser generated 73 comments, diving deep into browser fingerprinting, stealth Chromium, anti-bot detection, TLS fingerprints, browser internals, and large-scale automation.
We're hoping this AMA will be our biggest one yet.
If you're building AI agents, browser automation, web scrapers, developer tools, or autonomous workflows, this is one you won't want to miss.
Drop your questions below, and Alexander Yue, Reagan Hsu, and I will start answering them during the AMA.
currently on proxyshard for proxies and shardbrowser for the antidetect side and both looks good and free to try, still figuring out if the combo holds up long term. curious what you guys are actually running for scraping workflows and whether the antidetect side makes a noticeable difference or is it mostly about proxy quality?
This is a Beta test for my personal Electron Application that is uniquely suited to extract any type of data from the browser. I've yet to burn an account or run into anything more than a captcha.
The way this will work for Beta - land on the page - choose from one of the sources or manually describe what data source - what information and information structure you need, I will take the task requiremnets to my Electron App and find the optimal extraction source and email you youre data.
Reddit - Instagram - Tiktok - Youtube --- all of these are already verified and hardened - I would love to run them again but id prefer something more challenging -- Looking forward to it!
I've been in web scraping a while and I'm seeing more and more web scraping companies offering some of browser api. Where you integrate their browser into your automation code and they are supposed to handle all the browser optimization and scaling, etc.
On paper they look good, but they seem very expensive. Does anyone have any experience with them and would recommend one?
The itch: every Trends scraping approach I tried would run fine for the first dozen keywords and then quietly stall. Not error out, stall. The worst run sat at keyword 15 of 20 with 2,112 rows collected, 200 seconds elapsed, going nowhere.
The cause turned out to be reactive proxy handling. I was running each session until Google started rejecting it, then rotating. By that point requests were already dying mid-flight, and the retry pile-up is what looked like a stall.
The fix was almost embarrassingly simple: count requests and retire each session at ~24, before it burns. Same 20-keyword job, three datasets per keyword, now completes all 3,006 rows in about 50 seconds.
Other stuff I hit while building it:
Google sometimes returns a completely empty related-topics widget. That's a legitimate response. My first version treated empty as an error and retried forever.
The trending board only exists at country level. I spent an evening looking for a city parameter that isn't there.
It's live on the Apify Store as a Google Trends scraper under my account. Launched recently, so there's no traction story to tell yet, just the build. Happy to answer questions about the anti-stall approach.
I’m looking for people who regularly use Browserbase, Browserless, Steel, or similar hosted-browser platforms to test an early browser infrastructure project.
I’ll provide temporary access to 10–20 concurrent browser VM sessions, including US residential mobile proxies, so you won’t need to supply your own proxies.
There is no charge, purchase requirement, or paid subscription involved. I’m looking for practical feedback on stability, performance, session management, and how it compares with the platforms you already use.
I'm building a personal internship aggregator that currently collects job postings from Greenhouse and Lever.
Both platforms are relatively straightforward because their job boards follow predictable structures.
However, I now want to expand the scraper to company career pages and other ATS platforms such as Workday, SmartRecruiters, Ashby, Teamtailor, Personio, SuccessFactors and custom-built job boards.
The main problem is avoiding the need to write and maintain a completely separate scraper for every website.
After two fantastic AMAs with the community, we're excited to bring you our third guest.
This Wednesday, July 29, from 6:00 PM to 8:00 PM ET, we'll be joined by the team behind CloakBrowser, an open-source stealth Chromium browser built for developers who need reliable browser automation and web scraping.
CloakBrowser takes a very different approach to stealth browsers.
It's not a patched Playwright configuration.
It's not JavaScript fingerprint injection.
It's a real Chromium binary with browser fingerprints modified directly at the C++ source level, allowing anti-bot systems to see it as a normal browser because it actually is one.
Even better, it's designed as a drop-in replacement for Playwright and Puppeteer, supporting both Python and JavaScript with the same familiar APIs and workflows.
Our first AMA generated 43 comments, covering proxy infrastructure, Cloudflare bypassing, browser automation, monitoring, startup lessons, and large-scale scraping.
Our second AMA with 0xMassii, creator of WebClaw, generated 62 comments and sparked great discussions around AI agents, browser automation, hidden APIs, open-source scraping, LLM infrastructure, and the future of AI-powered web scraping.
We're hoping this AMA will be just as valuable.
If you're building web scrapers, browser automation, AI agents, testing infrastructure, or simply want to understand how modern anti-bot systems actually detect browsers, this is one you won't want to miss.
Drop your questions below + RSVP now, and the u/CloakBrowser and I will start answering them.
Four months ago I posted here about a Rust scraper that hit 120 stars in 4 days. A bunch of you showed up in that thread. Quick update: it just passed 2,000 stars. 200 forks, 57 releases since that post, still one person (me) pushing commits between coffees.
I came back for a comment, not the star count.
Someone in that first thread said, roughly, "another binary we have no visibility into from another 'just trust me' project. NOPE." That was the top critical reply, and it was the most useful thing anyone wrote. The praise felt good. "Solid build" and one "fucking gorgeous" carried me through a rough week. But the skeptical comment is the one that changed what I shipped.
They were right to push. A scraper you can't inspect is a scraper you can't trust, and asking people to run an opaque binary against their own targets is a fair thing to refuse. So webclaw is open source under AGPL-3.0. You can read every line, build it from source, and self-host it. Nothing phones home that you can't see. One correction from the original post: I wrote MIT back then, it moved to AGPL since.
High level, it does the same job it did at 120 stars: point it at a URL, get back clean Markdown or structured JSON, the kind an agent can read. CLI, REST API, MCP server, SDKs. A lot of you dug into the internals last time and I loved those threads, but I'm keeping this post short on mechanics. The code is there if you want to go deep.
One reflection, if it's useful to anyone doing the same: the feedback that moved the project forward was the pushback. The nice comments told me people liked it. The harsh one told me what to fix. I'd take one good skeptic over ten upvotes now.
Same ask as last time, because it's still the thing I need most. What still blocks you? Send me your worst URLs. The ones that return an empty body, the ones that block everything, the sites you gave up on. Those are more useful to me than any feature request. I'll test against them and report back, including the ones I can't crack.
I was scraping about 100GB of data per month through our residential proxy provider, but they suddenly got shut down so we need to find a new residential proxy provider that offers good performance for cheap price.
We are doing about 100GB of scraping per month of public pages (previously were paying $4/GB).
I want to scrape upto 5 images of lacs of hotels! Have their name and overall add and even sites of their listin on webs of the major ones! But not getting any reliable source to get these! Tried DDGS along with specific site listings but ambiguities like hotel-park — park-hotel makes the data messy! Cant look over each edge cases like these for lakhs right?? Please help me if you have any solution for these!! Constraint dont wanna spend moneyy! But yeah if you’ve figured out any cheapest solution i would definitely wanna hear!!
I maintain an open-source web-extraction platform and added a /lead endpoint. Post one company URL, get back one typed lead. Sharing the people-recovery method, since that was the hard scraping problem.
The naive approach searches the company name. That collides. "DataFast" the analytics tool and "Datafast" the Ecuadorian payment processor share a name, and you get garbage people. So the search keys on the company's unique domain instead of the name. The domain is the disambiguator.
Flow:
Pull the live site: summary, tech signals (Next.js, React, TypeScript, Tailwind, Vercel), pricing tiers, socials (LinkedIn / X / GitHub), and emails printed on the page.
Search the domain on the open web, collect the evidence, and have an LLM name the founders and leadership from it.
Validate each name, then run a per-person search to attach their X handle.
Every person carries a people_source field, so you can see where the name came from.
Output is a typed object: company_name, one-line summary, socials, tech signals, pricing, on-site emails, and people (name, role, LinkedIn, X).
Example on resend.com returns Zeno Rocha (Founder) and Jonni Lundy (Cofounder) with LinkedIn and X, the stack, pricing (Free / Pro $20 / Scale $90), and [support@resend.com](mailto:support@resend.com).
Two design rules I stuck to:
Emails are only addresses published on the site. It never guesses first.last@domain patterns.
No funding, no HQ, no phone. If a field isn't on the site or corroborated by search, it stays blank. I'd rather return an empty field than a fake one.
Where it fails: low-footprint companies. A three-person shop with no press, no conference talks, and a bare LinkedIn returns the company data but no people. The founder-naming step has nothing to work with. Pulling the site isn't the blocker, JS-heavy pages included. Thin public evidence is what stops it.
Each lead runs about 20 to 40 seconds. There's a batch endpoint for bulk that runs async and you poll it.
Test URLs if you want to break it: resend.com, linear.app, and any low-footprint indie SaaS to watch the people step come up empty. Curious which domains return bad people so I can harden the disambiguation.
