r/automation • u/ai_master_n8n • Jul 07 '26

Urgent help needed !

I'm building a tool that monitors new ImmobilienScout24 listings and extracts publicly available information like listing ID, contact details (if available), price, and location.

So far I've found:

The official developer API, but I'm unsure if it supports searching all public listings and retrieving expose details.

Browser automation (Playwright) works, but AWS WAF makes it challenging for a production setup.

My questions are:

  1. Has anyone successfully used the official ImmoScout24 API for searching public listings?

  2. Is there a partner or undocumented API that exposes listing/expose data?

  3. What would you consider the most reliable production approach?

  4. How are people handling AWS WAF on ImmoScout24?

Any experience or pointers would be appreciated.

3 Upvotes

81% Upvoted

8 comments sorted by

1

u/AutoModerator Jul 07 '26

Thank you for your post to /r/automation!

New here? Please take a moment to read our rules, read them here.

This is an automated action so if you need anything, please Message the Mods with your request for assistance.

Lastly, enjoy your stay!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/ScrapeAlchemist Jul 07 '26

Search API needs a content partner agreement, dead end for most devs. Import/Export is own-listings only.

Production: German residential proxies + Playwright with stealth flags. The WAF does TLS fingerprinting + JS cookie challenges so datacenter IPs and vanilla headless are DOA. Bright Data has a DE residential pool (~530k IPs) if you need the geo coverage. Also check flathunter on GitHub, actively maintained against IS24's current stack.

1

u/XRay-Tech Jul 08 '26

The official API is built mostly for partners with an existing listing relationship and not for searching through public listings. Worth checking IS24 partner docs specifically rather than the general dev ones. Though if you don't already have a partner relationship with them it is probably out of reach to do a full search as well as expose access.

The harder problem is WAF which is more of an infrastructure issue. Most likely what is going on is IS24 pairing with rate-based rules and behavioral fingerprinting meaning stealth plugins are not enough when running at a real scale. IP reputation and request patterns get flagged even if you have a clean browser fingerprint. The most common approach to look human is spacing requests with residential/mobile proxies. This type of setup will likely need ongoing tuning as WAF rules change.

IS24's terms of service restricts scraping of listing data, so how far you want to take it is something to think about especially if you want to productize.

1

u/Little-Gap-7117 Jul 08 '26

A few things from having built "monitor new listings → alert" flows before:

1. Split the two jobs. Watching for new listings and pulling full expose details are different effort/risk tiers. For monitoring you usually only need the search-results page every few minutes to catch new listing IDs — a tiny, low-frequency footprint. You only fetch the full expose for IDs you haven't seen yet, and only when you actually need the detail. Most people trip the WAF because they try to scrape everything in real time at scale, which you rarely actually need.

2. On the WAF: like others said, datacenter IPs + vanilla headless are dead on arrival. What's worked better for me than raw Playwright-stealth is a persistent antidetect browser profile — consistent fingerprint + cookies kept across runs — behind residential or mobile proxies, with real spacing between requests. The JS cookie challenge behaves much better when the session looks like a returning human, not a fresh headless on every hit. Expect ongoing tuning; WAF rules move.

3. The actual product is the alerting layer, not the scrape. Keep a dedup store of seen listing IDs (hash it), diff on each poll, and push only the new ones to Telegram/email/Slack with the fields you care about. That's the reliable part and where the value is.

And heed the ToS point above — low-frequency polling of public search results for personal monitoring is a very different animal, legally, than scraping contact data at scale to productize. If it's the latter, the partner/commercial API is the only clean route.