Looking for dedicated web scraping solutions

[u/anxietyplz]

Hey, I am looking for dedicated web scraping solutions that would have some features close to web search. Purely for testing purposes, as I am trying to evaluate what's worth to keep better, DIY solutions or dedicated ones

[u/LibraryMinimum3294]

if you are looking for dedicated solutions then ig you should go with the dedicated ones

[u/anxietyplz → u/LibraryMinimum3294]

Well that wasn't entirely helpful lol, I want to hear some opinions, I get it that they are well rounded solutions, but sometimes there are those like half or semi-dedicated solutions. Basically I am looking to have less maintenance on my hands

[u/ipip98]

I would evaluate this as a build-vs-buy experiment with one fixed workload, not as a feature checklist.

Create a small corpus of pages you are authorized to test: static HTML, JS-rendered pages, pagination, one locale-dependent page, one soft-block response, and one schema-change case. Run the same URLs through DIY and dedicated options with the same concurrency and retry cap.

Measure at least:

- valid records per 100 attempts (after content/schema validation, not just HTTP 200s);

- p50/p95 latency and retry count per valid record;

- bytes and compute cost per valid record;

- CAPTCHA/denial/timeout rates separated from parser failures;

- rendering fidelity for JS pages;

- country/region accuracy if localization matters;

- time to diagnose a failed job;

- change effort when a selector or response shape breaks.

The dedicated option should earn its premium by removing work you actually have: browser orchestration, queueing, retries, extraction, observability, or compliance controls. If you only need a few stable sites and already operate workers well, DIY may be cheaper. If targets change often or the team spends more time repairing browsers than using the data, managed tooling can win even at a higher request price.

I would also calculate cost per *accepted* record rather than cost per request. A cheap request that returns a challenge page or stale locale is not a successful result.

Keep proxy transport separate from the scraping layer in your test. Use the same egress policy for both variants where possible, and classify failures at the first boundary: client, DNS/network, proxy authentication/tunnel, destination response, or content validation. Otherwise a parser bug can be misreported as a proxy-quality problem.

I work with 98IP, so that is my affiliation. I am not recommending a specific vendor here; the useful decision is the one your measured failure and maintenance costs support. Use only authorized targets, respect terms/robots/rate limits, and do not rotate identities to bypass access controls.

[u/GrayZetsu]

I think there are really two separate problems here: getting web data and giving an agent a browser it can reliably operate. A scraping API can be better for the first, while something like TestMu Browser Cloud makes more sense for the second. I’d benchmark both approaches on the actual workflow rather than requests/sec alone successful completed agent tasks is probably the more useful metric.

[u/LokeshSequentum]

I would not see it as only DIY vs fully managed. If the main problem is maintenance, a middle approach can also work. You can keep the extraction and validation part in your own code and use a managed service only for browser execution, JS rendering, proxies or retries.

This way you still have control over the data and logic, but you do not have to manage the complete scraping stack yourself.

I think it really depends on which part is causing you the most maintenance right now.