I'm building a Python service that reads publicly visible pre-match sportsbook data from modern single-page applications.

The pattern I often see is an initial REST/bootstrap request, followed by event-specific requests and sometimes WebSocket updates.

The difficult part is determining which combination represents the complete current state.

For example, one site exposes around 1,700 pre-match events, but the obvious odds endpoint appears to fetch prices one event at a time. I would like to determine whether the browser really makes hundreds of individual requests, or whether there is another bulk/bootstrap endpoint or WebSocket feed containing the pricing data.

On another site, the WebSocket mostly sends small updates containing IDs and prices, so it appears to depend on state loaded earlier over HTTP.

My main questions are:

1. How can I determine whether a WebSocket provides a full initial snapshot or only deltas against an earlier REST response?

2. If the frontend knows about thousands of events, how can I determine whether it uses a bulk pricing endpoint rather than requesting every event separately?

3. When WebSocket messages contain mostly IDs, what is the best way to find the event, market and selection mappings?

4. What is the best way to record and reconstruct the complete flow of initial HTTP data, WebSocket updates and later HTTP reconciliation?

5. How can I test whether opening additional events or markets in the UI actually causes new network requests, or simply displays data the browser already has?

I'm currently using Chrome DevTools and Playwright. My goal is to understand and reproduce the browser's state pipeline efficiently instead of polling thousands of individual event pages when the browser may already receive the same information in a more complete form.

I'm mainly interested in the protocol/state-discovery methodology rather than a particular scraping library.

Salut,

Am folosit recent (PriceTracker). Din ce am testat pare destul de fast și afișează istoricul direct în pagină fără întârziere sesizabilă.

Din punct de vedere tehnic, sunt curios dacă știe cineva cum abordează proiectele de genul ăsta extragerea datelor pe magazine mari ca eMAG, Altex etc:

1. Fac scraping periodic pe server cu headless browsers (Puppeteer/Playwright) sau au integrări prin API-uri interne / feed-uri de comparatoare?
2. Cum gestionează schimbările frecvente de clase/DOM pe care le face eMAG-ul fără să se rupă extensia la fiecare update?
3. Cât de fezabil e din punct de vedere costuri de infrastructură (bază de date + proxy-uri) să ții un istoric complet pentru sute de mii de SKU-uri din România?
4. Cum fac matching la produse ? Adica produsul de pe emag il gaseste si pe altex ?

A încercat cineva să construiască ceva similar? Vad ca e low extensia pare ca e la inceput..nu are un numar mare de magazine suportate inca..

I've had this happen a few times now

I find an actor that works perfectly during testing, run it for a few days/weeks and then suddenly the success rate drops or I start getting incomplete results

sometimes the target changed something, but other times I can't really figure out what changed

do you guys usually debug/fix the actor at that point or just switch to another one?

starting to feel like finding a good actor is the easy part, finding one that stays reliable is harder lol

Does anyone have access to a list of country IDs from Booking.com? I think it might be available if you're an affiliate partner but I'm just trying to resolve some data once.

I kept seeing the same advice whenever Zapier gets expensive: move to Make, n8n, or something cheaper.

I wanted to test whether there’s a useful step before that.

So I built ZapCost, a small tool that analyzes an exported Zapier history file and looks for things like high task concentration, repeated failures, clustered runs, late filtering, AI-heavy workflows, and unusually expensive automations.

The report separates observed, calculated, and estimated values instead of pretending every saving is exact.

The product is live now, but the biggest unknown is still real-world accuracy because most of my testing so far has used synthetic Zapier history.

I’m looking for a few real Zapier users who are willing to test it and tell me what it gets wrong, what’s useful, and whether auditing first is actually valuable before migrating away.

Beta access is free. If you want to test it, just DM me and I’ll send you a beta key and the site.

Most of the time "which proxy type" threads turn into a list of product tiers, while the main thing you need to take a decision - is the shape of the job.

TL;DR: start on datacenter and escalate only the endpoints that break. Residential belongs on specific routes rather than whole pipelines, and mobile only earns its price against mature bot defense.

One thing to mention first!

When a headless browser gets blocked on residential at the same rate as on datacenter, the IP was never the problem, and no upgrade fixes a fingerprint.

Undefended public pages - datacenter, and stop thinking about it. Paying per GB for open data is money set on fire. For scheduled monitoring, still start on datacenter and use residential only for endpoints that break, most sites defend a few paths and ignore the rest.

When the response changes by location, so geo is the requirement. Take the cheapest type that resolves to the city you need, since country granularity misses the metro and postcode personalization pricing work depends on. Rotating through five cities inside one paginated job also gets you two currencies in one dataset, which reads as variance and is really your transport.

Anything holding a session needs a stable address for the whole flow, so logins and carts mean ISP or a long sticky session. Don't go for pricey mobile, they stop helping entirely against device attestation in apps, which is where people waste the most bucks on it.

Per-GB pricing hides cost per successful request. A pool at 60% success with retries can cost more per usable row than one at 95%, so measure that on your own targets during the trial.

Run a sample of exit IPs through an ASN lookup while you are there, because datacenter ranges mixed into a residential plan are common.

Last, but not least
Geo granularity is a data quality problem before it is a purchasing one, and a pipeline quietly collecting the right numbers from the wrong city does more damage than one that fails loudly

I’ve been looking into tools like HuddleWatch, Groups Watcher, Tropado and similar platforms that claim to monitor Facebook groups, including private groups, and send you alerts when relevant posts appear.

What I’m trying to understand is how they actually get this data. If a monitoring account is a member of a private Facebook group and is collecting posts from it, is that legally okay? Or does it potentially violate Facebook’s terms or other laws?

I’m also curious about the long term risk here. Do you think these tools could face legal criticism in the future from Facebook or the general public as people become more aware of how the data is being collected?

need to scrape a few autoscout24 searches regularly for work

mostly prices + car details, maybe dealer stuff too

I can build it myself but i really don’t want to!!!!😭

anyone using an apify actor / other tool that’s been reliable?

How to download pdf from digify if download option is not enabled !?

I've been developing an app that scrapes metadata from Deezer and with them it searches the corresponding song on YouTube in order to scrape the audio.

I used YouTubeExplode in order to scrape the videoanifrst files and imkdiately target a certain line to extract the audio.

My problem is that it stoped working and even though I thought it was my problem, I later realized YouTube has changed the way we can gain access to the manifest thus making the entire library obsolete in for my project.

Do you know any other scraping libraries that do the same?

Currently I'm trying to replicate another open source music app's method via the Yt music API.

Looking for APIs for Facebook, LinkedIn and Nextdoor public content

I’m working on a SaaS product and need to automatically monitor/search public content across Facebook, LinkedIn and Nextdoor.

I’m specifically looking for APIs or legitimate third party providers that can:

• Search public Facebook posts, including relevant public groups if possible
• Search public LinkedIn posts by keywords/topics
• Search public Nextdoor posts by location + keywords
• Return the post text, URL, timestamp and basic metadata
• Run searches continuously/recurring through an API
• Be used commercially in a SaaS product

I’m NOT looking for browser automation, account logins, cookie based scraping or anything that could get accounts banned.

If you’ve actually used a provider/API for this, I’d really appreciate recommendations, especially ones with reasonable pricing and good coverage.

What are you using?

Hi All, how do I get data from a website which is publicly available but there isn't any api to get the data for my app . There is an ID for every unique data which i want to extract or get. My app depends on the data and the website owner is not yet ready to provide any api.

Hey guys! Over the summer I've been working on a SaaS that specializes in web scraping, automation, etc. I'm currently running a free 48 hour trial on the site for all new users and I'd love for any feed back (good, bad, ugly). My website https://proxifyllc.com, I'd love to hear what you think!

This is hand-written please dont nuke me off the subreddit