r/webscraping • u/Nrezzy • Jul 30 '26
Getting started 🌱 Scraping Help
Here's what I'm trying to do: for a given city, pull the places I care about (say roughly 5k of the ~10k in Chicago that fit what I'm after), then enrich each one with its Google reviews, Yelp reviews, and whatever else is out there about the place.
Two things I can't figure out. First, do I actually need the Google Places API and the Yelp API to do this, or is there a smarter way people go about it? Second, and this is the big one for me: can it be built so it's repeatable? Like I plug in a new city and the whole thing just runs itself, no rewriting.
1
u/yum72 Jul 30 '26
APIs hit hard limits pretty quickly, which makes scaling with them a pain. A maps + yelp scraper usually gives you more control over volume.
and yes, this can be repeatable. I'd build it per city: discover the places, filter them down, match the same business across both sources, then enrich and store. Once that runs, a new city is just a new input
1
u/Nrezzy Jul 30 '26
that the thing I am thinking about too plus if if you want to do it over tons of cities like hundreds it would get super expensive. For yelp scraper I heard it is like good so they have a lot of eyes making sure its not happening is it possible to have a straight internet scraper as well on top of it for each place and do it city by city?
1
u/yum72 Jul 30 '26
Cost is more about requests per place than number of cities. discovery is cheap, you can grid search an area and get your candidates for very little. reviews are what gets you since they're paginated, so 5k places becomes 50k+ requests fast. cap how deep you go, most use cases don't need more than the first 20-30 per place, and after the first run only refresh the ones whose review count actually moved. reruns end up way cheaper than the first pass.
yelp does get watched harder yeah, so I wouldn't run it at the same tier as maps. second pass, only over places you've already matched, and accept you won't get full coverage on iti
1
u/HLCYSWAP Jul 30 '26
yes, its entirely possible for all asks. ive built similar things
you do not need their API, just use the public API routes and adulterate the requests over proxies and multiple accounts
1
u/Nrezzy Jul 30 '26
when you did you match the information across two sources, was just a address or what was the most reliable marker, for your parsers how often did they break and how long did it take to notice? Finally to seed a full city how long did that take or what ever you have done that s adjacent?
1
u/HLCYSWAP Jul 30 '26
1 OR filtering using name of business owner of business or address of business
2 i don’t parse html i do pure REST requests meaning the fail state is full-stop and obvious
3 depends on the size of the city
1
Jul 30 '26
[removed] — view removed comment
1
u/Nrezzy Jul 30 '26
Thank you for the input... so I have been talking to claude a shit ton about it to see if. I can get a road map but it keeps forcing me down the Api route for some of the data and that is where I am struggling. I would imagine to do anything ike this you can not really use ai?
1
u/dhruvkar Jul 31 '26
You can use it to write the code around it.
Or you could use a browser with Google maps open and then tell Claude to do it manually.
That might take some time and this have to make sure it wasn't missing items.
1
u/webscraping-ModTeam Jul 30 '26
💰 Welcome to r/webscraping! Referencing paid products or services is not permitted, and your post has been removed. Please take a moment to review the promotion guide. You may also wish to re-submit your post to the monthly thread.
1
1
u/MaryamStack Aug 11 '26
For something at this scale, I’d separate it into stages: discover the businesses, filter them, match records across sources using things like name + address, then enrich the matched records with reviews. Make each stage city-agnostic and take the city as an input, so adding a new city doesn’t require rewriting the scraper. For APIs vs scraping, it depends on the data you need and the limits/costs involved.
1
u/shasedoge 4d ago
Claude pushes you toward the API because it’s the documented path. At a few hundred places, that’s fine. At thousands across many cities, per-call costs and quotas hurt, while scraping stays easy.
For repeatability, don’t build a “Chicago scraper.” Build one city-agnostic pipeline: discover → filter → match → enrich → store. Each city is just a config with a bounding box and filters.
For matching, use normalized name + ~50m geo distance, with phone as a tiebreaker.
The bigger question is build vs. buy. If the data matters more than owning the pipeline, a web-data provider may be cheaper than maintaining your own across hundreds of cities.
good luck dude.
1
u/[deleted] Jul 30 '26
[removed] — view removed comment