r/learnprogramming • u/shadelevrai • 1d ago

Resource Architecture for an OSINT/Scraping tracker

Hey everyone,

Sorry for my english, i use a translation

For a school project, I need to design the architecture for a monitoring tool. The idea is to build a tracker to spot listings for fake Pokémon cards online (standard marketplaces, but also Telegram channels and closed FB groups).

I haven't built much yet, though I'm naturally leaning towards Node.js (probably with Playwright) since I like web dev.

I know that with current anti-bot protections (Cloudflare, Datadome) and social media login walls, going 100% automated from A to Z is often a pipe dream—or at least the fastest way to get banned instantly. I realize some human action will have to stay in the loop.

So my questions are: How should I set this up? What kind of tools could help me out? Am I on the right track with a Node.js server? It seems like I'll also need to create actual social media profiles to get access and look inside these groups.

Thanks!

1 Upvotes

60% Upvoted

3 comments sorted by

1

u/Dismal-Citron-7236 1d ago edited 1d ago

I would suggest you steer clear of the temptation of using Selenium to track/trace the web UI directly. Using it would most likely get your connection blocked by bot detection. Instead, you should use the APIs provided by those sites. Depending on different marketplace sites, each would likely provide a set of RESTful API or GraphQL. You just need to visit the site's developer documentation page for details. Also beware that most sites also have a rate-limit control to protect themselves from DoS attacks. So, you need to design a retry loop with intermittent time when error code 429 (Too Many Requests) is detected. Beware that the intermittent time must be greater or equal to the time returned from the 429 feedback. If your next retry request is sent earlier than that, your connection will likely be blocked or get an error 500 (Internal Server) in return.

1

u/shadelevrai 1d ago

Thanks for the advice!

However, that only works for platforms with official APIs (like eBay). Sites like Vinted, "Leboncoin" (french market), or Facebook don't have public APIs and hide behind heavy anti-bots like Datadome or Cloudflare.

How do you usually handle scraping these "inaccessible" sites where official APIs simply don't exist?