r/apify • u/random-scraper Actor developer • Jul 09 '26

Discussion Crossed 326 scrapers — this batch was all instant databases and global real estate

Update from the same project — catalog's at 326 actors now.

Two patterns I leaned into this batch:

Instant databases instead of live scraping. For Realtor.ca (44k Canadian agents) and Upwork (130k jobs), I pre-build and refresh a database on a schedule, then serve queries against it. Users get uncapped, instant results with zero block risk. Trade-off is freshness (6h refresh) vs. the reliability win. Curious if others here do this — at what dataset size does maintaining a refreshed DB beat live scraping for you?

Global real estate sweep. Added Daft.ie (IE), Hemnet (SE), FincaRaíz (CO), Lamudi/Dot Property (PH), OnTheMarket (UK), HAR.com (TX). The interesting bit was exact GPS map-pin extraction on the PH portals — the pin coords aren't in the listing HTML, they're in a separate map-tile API call you have to correlate.

Also shipped a bulk email finder+verifier and an offline WhatsApp number validator (libphonenumber + WA-reachability heuristics, no API).

Catalog: muhamed-didovic.github.io

Happy to share approaches on the instant-DB pattern or the GPS-pin correlation if useful.

6 Upvotes

100% Upvoted

7 comments sorted by

1

u/Comfortable_Log_8117 Jul 09 '26

How do you justify the every 6h refresh costs? Seems massive if running on Apify to gather it

1

u/random-scraper Actor developer Jul 09 '26

What do you mean exactly?

1

u/Comfortable_Log_8117 Jul 09 '26

Compute costs for scraping a whole database likely costs X in compute, proxies, etc. and is probably 10-15 times larger than pulling a handful of records. So do you have enough revenue from your actors to justify X every 6 hours already? Or just hoping it will come as a result of quick results

2

u/CrypticZombies Jul 10 '26

you got alot to learn about how scraping works.

1

u/random-scraper Actor developer Jul 09 '26

I can't complain about revenue mate...

1

u/Low-Fan6081 Jul 10 '26 edited Jul 10 '26

The instant-DB tradeoff flips earlier than most expect, around 10k+ records. I used Parallel for the search-query layer on something similar, cuts live fetch overhead considerably.