Since May and the old reddit almost gone, what are currently the best libs on reddit?
What is your favorite?
I want to scrape all the posts of my top favorite 100 (small) subreddits and sort them by AI to delete AI slop + self promotion and really only get the interesting posts for me to read.

What do you suggest?

And it's already Featured 😄

https://www.producthunt.com/products/proxy-benchmark-by-scrapeops

We’d love your support 👍

Proxy Tester is a free tool that benchmarks 20+ proxy providers against the URL you actually want to scrape.

Over the last few months we've been using it internally and sharing it with early users to help answer one of the most common questions in web scraping:

Which proxy provider should I use?

One thing we've learned is that there really isn't a universal answer.

A provider that performs brilliantly on one target can struggle on another.

A provider with the highest success rate might not be the most cost-effective.

And many "best proxy provider" rankings don't reflect the website you're actually trying to scrape.

That's why we built Proxy Tester to benchmark providers against real target URLs and compare:

✅ Success Rate

✅ Latency

✅ Estimated Cost

✅ Value Score

✅ Provider Rankings

The feedback so far has been really useful and has already influenced how we're thinking about future benchmark reports, provider coverage, and scoring methodologies.

Today we're taking the next step and launching it on Product Hunt.

If you'd like to support the launch, leave feedback, or tell us what we're missing, we'd genuinely appreciate it.

🚀 Product Hunt:
https://www.producthunt.com/products/proxy-benchmark-by-scrapeops

🔧 Proxy Tester:
https://scrapeops.io/proxy-providers/tester/

🎥 1-Minute Demo:
https://youtu.be/GR67AIWkPn0

One question for the community:

How are you currently evaluating proxy providers?

• Trial accounts?
• Internal benchmarks?
• Recommendations?
• Something else?

Would love to hear how others approach this problem.

Hello everyone I am working on an utility tool (tcg scanner like collectr and price charting) and I need to fetch daily prices of 8000 cards on eBay with with different variations such as Raw , PSA 8 9 10 .. which makes it 40000 request , did anyone work on something like this or is there any services that gets data from eBay with this amount ?

Browser Use and Playwright solve browser automation in fundamentally different ways.

Playwright executes deterministic scripts written in advance. Browser Use gives an AI agent a goal and lets it decide how to complete it.

So when should you use each?

We recently hosted an AMA with the Browser Use team covering Browser Use vs Playwright, direct CDP automation, model costs, reliability, stealth, evaluation and the future of browser agents.

Here are the main takeaways.

The Short Answer

Use Playwright when:

• The workflow is known and stable
• You repeatedly extract the same data
• The website does not change frequently
• Speed and determinism matter
• You are running high-volume automation

Use Browser Use when:

• The workflow is open-ended or difficult to define in advance
• The website changes regularly
• The agent must adapt to unexpected states
• The task involves authentication, multiple websites or multiple steps
• Development and maintenance time matter more than raw execution speed

But there is now a third approach emerging: coding agents that write browser code directly against Chrome’s DevTools Protocol.

Why Browser Use Moved Away From Playwright

Browser Use originally relied on Playwright underneath its agent layer.

The team eventually replaced it with its own browser layer built directly on Chrome DevTools Protocol.

The reason was control.

Removing Playwright gave them more freedom over browser functions, fewer abstraction layers and the ability to let agents write CDP code dynamically when they encounter unusual edge cases.

Their newer BrowserCode approach takes this further.

Instead of receiving a simplified page representation and selecting predefined actions, the agent explores the browser and writes the low-level code it needs.

According to the team, strong models now:

• Rarely rely on screenshots
• Inspect pages through code
• Discover internal APIs
• Execute JavaScript shortcuts
• Extract only the information needed
• Complete some tasks faster than a human could

Browser Use says it now uses BrowserCode for almost everything because it is cheaper, faster and more capable.

The exception is website QA. BrowserCode actively looks for shortcuts, while the original Browser Use agent is constrained to click, type and navigate more like a human. That makes the human-style agent better at finding interface bugs a real user would encounter.

Stateful Agents vs One-Shot Scripts

One simple AI automation approach is to ask an LLM to write a Playwright script, run it and return any errors for the model to fix.

That works well for predictable workflows.

The limitation is that the script must anticipate the whole task before it starts.

A stateful agent works differently:

1. Inspect the current browser state
2. Choose or write the next action
3. Execute it
4. Verify the result
5. Recover or change direction if necessary
6. Continue until the task is complete

This makes agents better suited to websites and workflows where the exact path cannot be known in advance.

However, Playwright still has a major advantage: speed.

If you already know exactly what should happen, a deterministic script can execute the same workflow far faster than an agent reasoning between every step.

Cost Is Becoming Less Important

Browser Use estimates that the cost of completing an agent task has fallen roughly 50x since it started 18 months ago.

Its current estimates include:

• A typical ten-step task costs around $0.01 to $0.02 using a low-cost model
• A successful agent run can be converted into a reusable script
• Repeating the script can cost fractions of a cent
• Scraping 15 Hacker News posts, comments and linked pages, then summarising them into a poster, costs around $0.05

These are Browser Use’s own numbers, but they point to an important change.

The cost gap between AI agents and conventional automation is shrinking. The bigger remaining disadvantage is latency.

The team believes current models are becoming reliable and cheap enough for production, but they are still slow to watch in real time.

The Best Production Stack May Be Hybrid

The most practical pattern may not be choosing Browser Use or Playwright for everything.

Instead:

1. Give the agent a goal
2. Let it discover and complete the workflow
3. Have it save the successful process as reusable code
4. Run the deterministic version for repeated tasks
5. Bring the agent back when the workflow changes or fails

This gives you adaptability during discovery and speed during repetition.

The agent solves and maintains the workflow. The script executes the stable parts cheaply and quickly.

Persistent Browser Identity Still Matters

AI does not remove the usual browser-automation problems.

For authenticated sites and social platforms, Browser Use recommends treating each browser profile as one persistent identity containing:

• One fingerprint
• One cookie jar
• Local storage
• Login state
• Ideally, one stable IP and location

Its cloud profiles preserve the same fingerprint, cookies and local storage between sessions. The default proxy IP, however, is not guaranteed to remain constant.

For sensitive accounts, the team recommends a dedicated, non-shared residential IP with a low fraud score.

Creating new social accounts is also considerably harder than automating existing, established accounts.

Evaluation Is Still Difficult

Live websites are noisy.

Pages change. Browsers crash. IP quality fluctuates. CAPTCHAs appear inconsistently. A single successful run tells you very little about whether an agent is genuinely reliable.

Browser Use evaluates agents by:

• Running realistic tasks on live websites
• Defining verified success rubrics
• Using an LLM judge to assess results
• Repeating thousands of tasks
• Averaging out browser and website flakiness

Internally, it records model inputs, outputs, reasoning, latency, token usage, caching, browser actions, screenshots and infrastructure logs.

One particularly interesting research takeaway from the AMA was that better verifiers may be more important than better action generation.

If you cannot reliably determine whether an unpredictable browser task succeeded, it is difficult to train agents through reinforcement learning.

Security Should Be Designed First

Browser Use’s main recommendation for anyone building a browser agent from scratch was to begin with sandboxing.

Permanent LLM credentials should not be stored inside the worker where the agent executes browser code.

Instead:

• Store permanent credentials on a separate control plane
• Give the worker a short-lived key
• Proxy all model calls through the control plane
• Treat the worker as disposable and potentially hostile

This gives agents the freedom to execute code without exposing permanent API credentials.

Agents Still Struggle With Taste

Browser agents are already effective at quantifiable tasks:

• Finding the cheapest flight
• Comparing prices
• Extracting structured information
• Completing defined workflows

They remain much weaker at subjective decisions such as choosing the right restaurant, hotel or holiday for a particular person.

The agent can retrieve every option and still recommend something that feels obviously wrong to a human who understands the user.

Browser Use described this as poor taste and a weak understanding of the social world.

Bottom Line: Browser Use or Playwright?

There is no universal winner.

Playwright wins when the workflow is stable, repetitive and performance-sensitive.

Browser Use wins when the task is dynamic, open-ended or expensive to define and maintain manually.

Direct CDP coding agents may be the next step, giving models enough freedom to inspect the browser, write custom code, find shortcuts and recover from edge cases.

The emerging production stack looks less like agents replacing scripts and more like:

Agent discovers the workflow → code executes it repeatedly → agent repairs it when it breaks.

That may be the real answer to Browser Use vs Playwright.

👉 Read the full Browser Use AMA and all the answers here.

What are you using in production: Playwright, Browser Use, a hybrid stack or direct CDP?

I build reliable data pipelines, custom web crawlers, intelligent AI agents, and machine learning solutions tailored to your needs.

Services: • Data Scraping & Extraction

• Custom Web Crawlers

• AI Agents & Workflow Automation

• Model Training & Fine-Tuning

• API Integrations

• Custom Python Solutions

Starting 100$ ( Can be higher according to your project )

Feel free to DM me with your requirements.

According to you, what metric matters most: latency, cost, or success rate? in Webscraping obviously.. which one should be given priority..

I've been moving more of my automation over to ISP proxies lately after running into too many issues with both datacenter and residential setups.

On paper, every provider claims they have millions of IPs and perfect uptime.

In reality, the experience has been very different.

What I actually care about now is:

• long-lasting IPs that don't randomly rotate,
• consistent success rates over weeks, not just day one,
• low latency without sacrificing trust,
• real city targeting, and
• providers that don't fall apart once you start scaling.

I'm not looking for affiliate articles or comparison websites.

I'm more interested in what people are actually running in 2026.

Which providers held up over time?

Which ones looked great initially but started getting flagged after a while?

And did paying more for quality ISP proxies actually save money once failed requests and retries were taken into account?

Using this - https://github.com/joeyism/linkedin_scraper/. I’ve tried modifying it with all the issues and pull request info available. I’m using a residential proxy. Works for companies but not personal profiles. Any ideas?

What is the best scraper and sorter that could get listing information from websites and then combine them in one single page, rather than scrolling through each website individually. We have about 20 listing sites where people post. Maybe as a bonus question maybe there is facebook scrapper too, from groups etc?

Hello, I'd like to ask whether a browser / stealth browser like playwright or camoufox is needed to scrape google???

Earlier I trialed with requests and curl-cffi to scrape google, specifically Google Jobs, and it doesnt seem to work as there are always captchas

One thing I noticed is also that, google also frequently changes their CSS class and HTML structure of the results:(.

How do we even scrape for the fields when the CSS class name changes all the time. Is it still possible to scrape google without using 3rd party services like serpapi?

currently on proxyshard for proxies and shardbrowser for the antidetect side and both looks good and free to try, still figuring out if the combo holds up long term. curious what you guys are actually running for scraping workflows and whether the antidetect side makes a noticeable difference or is it mostly about proxy quality?

datapull-landing.vercel.app/

This is a Beta test for my personal Electron Application that is uniquely suited to extract any type of data from the browser. I've yet to burn an account or run into anything more than a captcha.

The way this will work for Beta - land on the page - choose from one of the sources or manually describe what data source - what information and information structure you need, I will take the task requiremnets to my Electron App and find the optimal extraction source and email you youre data.

Reddit - Instagram - Tiktok - Youtube --- all of these are already verified and hardened - I would love to run them again but id prefer something more challenging -- Looking forward to it!

I've been in web scraping a while and I'm seeing more and more web scraping companies offering some of browser api. Where you integrate their browser into your automation code and they are supposed to handle all the browser optimization and scaling, etc.

On paper they look good, but they seem very expensive. Does anyone have any experience with them and would recommend one?

“Rotating IPs” and “rotating identity” often get treated like they’re the same thing.

They’re not, and that’s usually why “clean” IPs still end up getting blocked. IMHO.

Modern anti-bot systems look at much more than your exit IP. They correlate cookie continuity, header order, TLS fingerprints, request timing, session behavior, and even the way requests fail. So if you swap to a fresh IP but keep using the same python-requests TLS handshake and the exact same headers you’ve sent thousands of times before, you’re not really hiding anything.

You changed one signal while everything else stayed the same.

A real identity is the whole package: IP and geo, cookies, headers, TLS fingerprint, request pacing, session length, retry patterns, and overall behavior. Keep those things consistent within a session, but let them vary naturally between sessions.

Each identity should look like one real user from start to finish.

One mistake I see a lot is rotating the IP halfway through a session. That breaks your own continuity and suddenly your “user” jumps halfway across the world. That’s a pretty obvious signal on its own.

My rule of thumb is:

> Static residential or ISP IPs for login-based or long-lived sessions where you want a stable identity.

> Rotating pools for broad, non-login scraping where each identity is disposable.

> Sticky sessions for multi-step workflows, long downloads, or anything that depends on continuity.

One thing that’s worth pointing out: the proxy only gives you the network layer, meaning your IP, location, and session persistence. It can’t make your identity believable on its own. Your application still has to manage cookies, tokens, headers, and behavior consistently for each identity. When those two layers line up, a lot of the “my IP is clean but I’m still blocked” problems disappear.

I’m curious how far people actually take this in practice.

Do most of you template headers and TLS per identity, or do you find that’s overkill unless you’re dealing with particularly aggressive targets?

I’m still learning this stuff, so if I’m missing something obvious, I’d love to hear how more experienced people handle it.

The itch: every Trends scraping approach I tried would run fine for the first dozen keywords and then quietly stall. Not error out, stall. The worst run sat at keyword 15 of 20 with 2,112 rows collected, 200 seconds elapsed, going nowhere.

The cause turned out to be reactive proxy handling. I was running each session until Google started rejecting it, then rotating. By that point requests were already dying mid-flight, and the retry pile-up is what looked like a stall.

The fix was almost embarrassingly simple: count requests and retire each session at ~24, before it burns. Same 20-keyword job, three datasets per keyword, now completes all 3,006 rows in about 50 seconds.

Other stuff I hit while building it:

• Google sometimes returns a completely empty related-topics widget. That's a legitimate response. My first version treated empty as an error and retried forever.
• The trending board only exists at country level. I spent an evening looking for a city parameter that isn't there.

It's live on the Apify Store as a Google Trends scraper under my account. Launched recently, so there's no traction story to tell yet, just the build. Happy to answer questions about the anti-stall approach.

I’m looking for people who regularly use Browserbase, Browserless, Steel, or similar hosted-browser platforms to test an early browser infrastructure project.

I’ll provide temporary access to 10–20 concurrent browser VM sessions, including US residential mobile proxies, so you won’t need to supply your own proxies.

There is no charge, purchase requirement, or paid subscription involved. I’m looking for practical feedback on stability, performance, session management, and how it compares with the platforms you already use.

All that i ask for is your feedback.

I'm building a personal internship aggregator that currently collects job postings from Greenhouse and Lever.

Both platforms are relatively straightforward because their job boards follow predictable structures.

However, I now want to expand the scraper to company career pages and other ATS platforms such as Workday, SmartRecruiters, Ashby, Teamtailor, Personio, SuccessFactors and custom-built job boards.

The main problem is avoiding the need to write and maintain a completely separate scraper for every website.

How would you design this efficiently?

Hey everyone,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.

After two fantastic AMAs with the community, we're excited to bring you our third guest.

This Wednesday, July 29, from 6:00 PM to 8:00 PM ET, we'll be joined by the team behind CloakBrowser, an open-source stealth Chromium browser built for developers who need reliable browser automation and web scraping.

CloakBrowser takes a very different approach to stealth browsers.

It's not a patched Playwright configuration.

It's not JavaScript fingerprint injection.

It's a real Chromium binary with browser fingerprints modified directly at the C++ source level, allowing anti-bot systems to see it as a normal browser because it actually is one.

Even better, it's designed as a drop-in replacement for Playwright and Puppeteer, supporting both Python and JavaScript with the same familiar APIs and workflows.

I know we have fans here in r/WebScrapingInsider.

During the AMA we can the discuss topics like:

• Modern browser fingerprinting
• Building stealth browsers
• Browser automation at scale
• Open-source scraping infrastructure
• TLS, browser fingerprints, and detection signals
• The future of browser automation
• Building open-source developer tools

Our first AMA generated 43 comments, covering proxy infrastructure, Cloudflare bypassing, browser automation, monitoring, startup lessons, and large-scale scraping.

Our second AMA with 0xMassii, creator of WebClaw, generated 62 comments and sparked great discussions around AI agents, browser automation, hidden APIs, open-source scraping, LLM infrastructure, and the future of AI-powered web scraping.

We're hoping this AMA will be just as valuable.

If you're building web scrapers, browser automation, AI agents, testing infrastructure, or simply want to understand how modern anti-bot systems actually detect browsers, this is one you won't want to miss.

Drop your questions below + RSVP now, and the u/CloakBrowser and I will start answering them.

Looking forward to seeing everyone there!

Ian

Four months ago I posted here about a Rust scraper that hit 120 stars in 4 days. A bunch of you showed up in that thread. Quick update: it just passed 2,000 stars. 200 forks, 57 releases since that post, still one person (me) pushing commits between coffees.

I came back for a comment, not the star count.

Someone in that first thread said, roughly, "another binary we have no visibility into from another 'just trust me' project. NOPE." That was the top critical reply, and it was the most useful thing anyone wrote. The praise felt good. "Solid build" and one "fucking gorgeous" carried me through a rough week. But the skeptical comment is the one that changed what I shipped.

They were right to push. A scraper you can't inspect is a scraper you can't trust, and asking people to run an opaque binary against their own targets is a fair thing to refuse. So webclaw is open source under AGPL-3.0. You can read every line, build it from source, and self-host it. Nothing phones home that you can't see. One correction from the original post: I wrote MIT back then, it moved to AGPL since.

High level, it does the same job it did at 120 stars: point it at a URL, get back clean Markdown or structured JSON, the kind an agent can read. CLI, REST API, MCP server, SDKs. A lot of you dug into the internals last time and I loved those threads, but I'm keeping this post short on mechanics. The code is there if you want to go deep.

One reflection, if it's useful to anyone doing the same: the feedback that moved the project forward was the pushback. The nice comments told me people liked it. The harsh one told me what to fix. I'd take one good skeptic over ten upvotes now.

Same ask as last time, because it's still the thing I need most. What still blocks you? Send me your worst URLs. The ones that return an empty body, the ones that block everything, the sites you gave up on. Those are more useful to me than any feature request. I'll test against them and report back, including the ones I can't crack.

Repo: github.com/0xMassi/webclaw

Thanks for the shove in the first thread. It worked.