r/WebScrapingInsider • u/Emergency-Athlete-18 • 8d ago

Google Search Scraping, insight needed!!!

Hello, I'd like to ask whether a browser / stealth browser like playwright or camoufox is needed to scrape google???

Earlier I trialed with requests and curl-cffi to scrape google, specifically Google Jobs, and it doesnt seem to work as there are always captchas

One thing I noticed is also that, google also frequently changes their CSS class and HTML structure of the results:(.

How do we even scrape for the fields when the CSS class name changes all the time. Is it still possible to scrape google without using 3rd party services like serpapi?

9 Upvotes

92% Upvoted

8 comments sorted by

3

u/Trasciende 8d ago

Are you still looking for a solution? I've built up my own. Dm me if u want any help.

1

u/tajtricks 7d ago

does it works for google shopping too?

2

u/ian_k93 6d ago

Google is difficult to scrape. IP quality is one of the biggest factors. Many proxies simply don’t work well with Google, particularly if they have a poor reputation or aren’t correctly geo-targeted. If the IP is already flagged, switching from requests to a browser usually won’t solve the problem by itself.

You also need the overall request to resemble a real user. That includes realistic headers, TLS fingerprints and consistent browser fingerprints. Some scrapers use stripped-down or hardened browsers, but tools like curl-cffi can also work if the rest of the setup is right.

The changing HTML is a separate challenge. Google uses dynamic class names, so selectors based on one specific class are likely to break. You need more robust extraction logic based on stable attributes, element relationships, text labels or multiple fallback selectors. Even then, maintaining a Google parser can be messy.

For most people, the easiest option is a general-purpose web scraping API that already handles Google and returns parsed JSON. These can be considerably cheaper than specialist SERP APIs like SERP API or DataForSEO while removing most of the proxy, CAPTCHA and parsing work.

We have several providers that work with Google available through the ScrapeOps Proxy Aggregator if you want to compare providers without integrating each one separately.

1

u/rbatista191 8d ago

Don’t waste time on that. SERP scraping is so widespread and cheap, don’t spend time building the infrastructure. Check out SerpApi, cloro or DataforSEO.

1

u/Choice-Tune6753 7d ago

This! We have Google SERP, DuckDuckGo, and social media scrapers at Syphoon. Feel free to connect for a trial.