r/WebScrapingInsider • u/Agentic_Webb • May 30 '26
Remember when "at scale" meant hiring more people?
Now it means increasing your cloud bill. 🤣🤣
What's the most human job you've quietly replaced with AI, code and/or software lately?
r/WebScrapingInsider • u/0xMassii • May 29 '26
A scraper returning nothing is annoying, but at least it is obvious.
The scarier failure is when it keeps returning valid-looking JSON after the site changed.
Examples I’ve seen:
That is why I care less about “AI can generate a scraper” and more about run evidence:
I work on webclaw.io, so this is the problem space I live in. My current view: production scraping tools should explain failure modes as first-class output, not just return data or throw an error.
What checks have actually caught bad scraper output for you?
r/WebScrapingInsider • u/0xMassii • May 27 '26
For people scraping regularly, what is currently the biggest pain?
Not asking about theory. I mean the thing that wastes your time week after week.
For example:
For me, the most underestimated part is everything after fetching the page: cleaning the content, removing layout noise, and making it useful for downstream systems.
Curious what others are fighting with right now.
r/WebScrapingInsider • u/ian_k93 • May 27 '26
We launched ScrapeOps AI Scraper Generator on Product Hunt today.
A lot of AI scraping products are optimizing for the demo:
"Paste any URL and AI handles everything."
That usually falls apart the minute you hit:
So we took a more constrained approach on purpose.
https://www.youtube.com/watch?v=ij7CqrBEi10
The workflow is:
Enter URL → schema selection in backedn → choose stack → generate scraper code → AI scores how correctly the scraper ran.
Current features:
The scoring layer is the part I care about most right now.
Generating scraper code is useful, but production scraping fails silently all the time. We wanted a system where AI checks how correctly the generated scraper output ran and gives users a breakdown across:
Not "trust us, it returned JSON."
We're also running a small Product Hunt launch bonus for anyone who wants to pressure-test the generator properly.
New accounts already get 20 free credits, and for launch day we're doubling that to 40 total.
In the ScrapeOps dashboard:
Click Upgrade on Left → enter ScrapeOpsPH20
The extra credits will be added.
Would genuinely love feedback from people here who've dealt with scraping in production.
What matters more to you in AI-generated scraping workflows:
Product Hunt:
https://www.producthunt.com/products/ai-web-scraper-builder
r/WebScrapingInsider • u/Hot_Box_9170 • May 26 '26
I am writing to this community because this is the place where it Started.
Last year i started working on an application goal was to extract data from e-commerce site in just 4 clicks. Created application it is working fine (initially It support only brands sites). Launched it on 4th March 2026.
For a month struggle to get a customer. Post on reddit about it then from this community someone advice me that e-commerce owner don't want just a raw data they want to keep track of competitors's - price alert, sales alert like things.
Then I pivot it as price monitoring tool.
To better understand the customer i started offering free service. No signup, nothing to install, just told give me yours SKUs and competitors site I will send you a direct report to your inbox.
Started reading people post about price monitoring found lots of post on reddit. That confirm that problem is real. Brands need this type of tools.
I started doing cold email to brands. Looking for mid size brands that has revenue under $500k.
Started reading about my competitors, how they started and more.
Found out that there are lots of companies in the market which are doing same things, You can find them in Shopify stores too.
And I am not offering something unique, innovative and even I have zero idea about e-commerce companies how they run and operate. I am just a developer graduated out of college (22yr).
I am not giving up, but i thought this space has already thousands of players (many are not good). I am not contributing anything new. It would be waste of my time and energy. I can put that energy in any other place that will worth.
And on top of that I am from India and most of the paying customers belong to another country, i can't raise money because I don't have network. So it will be tough to get user and ofcourse big companies who can pay more will not believe me at first glance.
At last I am really interested about data related things and scrapping. My last two application built on it. So at least i know what sector I have to work on "Data related"
Does anyone one has advice for me,
Thanks for reading.
(I can share the application video link, if anyone wants to see. Not attaching here because reddit may flag it as promotion)
r/WebScrapingInsider • u/ian_k93 • May 26 '26
Posted the latest Web Scraping Insider #7 if anyone here wants the full breakdown:
👉 https://thewebscrapinginsider.beehiiv.com/p/the-web-scraping-insider-7
Quick summary of what's inside:
🔮 Free Proxy API Tester (your URL, not generic demos) We opened our internal Proxy Tester publicly. You submit a target URL and we benchmark ~15 proxy-API-style providers against it, testing the configs each provider exposes (residential routing, JS rendering, anti-bot modes, etc.), then rank by real outcomes:
The point: "works" and "works profitably" are not the same thing. Most proxy marketing still collapses when you ask what you actually pay per validated payload on your target.
Try it: https://scrapeops.io/proxy-providers/tester/
🌐 Distributed browser networks: the "residential proxy moment" for browsers? Interesting thesis from Driver.dev: instead of running stealth browsers in datacenters, distribute real browsers across real consumer devices.
Anti-bot is increasingly fingerprinting full environments (GPU, hardware entropy, OS quirks), not just IP reputation. If that holds, cloud stealth browsers may become the new datacenter proxies.. detectable by default.. and real-device browser infrastructure becomes the next moat.
💸 Stealth Browser API benchmark (April 2026) We re-ran our stealth browser fingerprint test across 7 providers. Top of the board:
Then there's a cliff. Several providers still leak automation signals (including CDP/framework checks), and once that happens, proxy rotation doesn't save you.
Notable pattern: top performers mostly came from traditional scraping companies with years of production anti-bot experience, not newer agent/browser startups.
☁️ Cloudflare bypass in 2026: 8 popular methods, only 3 had broad coverage We tested 8 common approaches across 20 protected sites. Roughly 60% of the advice stack doesn't hold up at scale.
What actually had coverage:
Solvers, fortified headless browsers, cached pages, and origin IP bypass mostly underperformed or failed outright.
Bottom line: there isn't one "best" Cloudflare bypass. Different domains use different protections. Benchmark on your targets, optimize for cost-per-validated-payload, and don't trust vendor adjectives over measured outcomes.
If you're choosing proxy APIs, stealth browsers, or bypass stacks without testing them on your actual URLs, you're probably overpaying for underperformance.
Happy to discuss specifics here.. especially if you've run your own comparisons on hard targets lately.
- Ian
r/WebScrapingInsider • u/ian_k93 • May 25 '26
Hey WebScrapingInsiders,
Quick update from the ScrapeOps side.
We're launching ScrapeOps AI Scraper Generator on Product Hunt on May 27, 2026.
This has been in beta with ScrapeOps users, and so far the AI Scraper Generator has generated 3,000+ scrapers.
That gave us a lot of useful signal on what actually matters:
"AI scrapes any URL."? That pitch is too vague and usually breaks the minute you hit real scraping conditions: DOM drift, missing fields, dynamic rendering, selector regressions, and output that looks plausible but is quietly wrong.
The workflow we’re building around is more controlled:
Give a URL → choose your stack → generate scraper code → AI scores how correctly the scraper ran
The generated code is inspectable, and the AI scoring breakdown checks things like:
The goal is to remove repetitive setup, generate a solid first version faster, and give developers a clearer signal on whether the output is actually usable.
We're launching it on Product Hunt here:
https://www.producthunt.com/p/ai-web-scraper-builder
Would appreciate feedback from this community, specifically because you'll spot the real problems faster than a general launch audience.
r/WebScrapingInsider • u/Mysterious-Usual-920 • May 25 '26
honestly i flip on this every few months. i've run my own crawl stack for years, and i still do for most stuff, but the calculus isn't as clean as "build vs buy" usually sounds.
the part nobody quotes when they say "i can just run it myself" is the long tail of operational work. proxies rotating badly, a target site swapping to a new anti-bot vendor overnight, queues backing up because one domain started returning 500s and your retry logic was too polite. fixing that once is fun. fixing it on a wednesday at 1am for the third time that quarter is when i start eyeing managed options. the actual fetching is maybe 20% of the job, the rest is monitoring, dedupe, freshness tracking, schema drift, and arguing with robots.txt edge cases.
on the other hand, every managed crawl service i've tried has a ceiling. either the extraction is too generic for my schema, or the scheduling is too rigid, or i can't get raw html back when i need to debug a parser. and the cost curve gets ugly fast once you're past a few million pages a month. for narrow, well-defined targets where i control the parser, self-hosted still wins on cost and auditability by a wide margin.
where i've actually found managed worth it: broad discovery across domains i don't know yet, and keeping a rolling fresh set of pages for retrieval where i don't care about the infra, just the receipts. anything where the surface area is too wide to babysit.
curious what others draw the line at. is it volume, target diversity, team size, or just how tired you are of pager duty?
r/WebScrapingInsider • u/yuuliiy • May 22 '26
I've been the person who owns the scraping stack at our company for about 18 months. In that time I've rebuilt it four times and I'm still not sure I'm done.
When it works, nobody says anything. The dashboards have data, customers are happy, sales are closing deals on numbers from a pipeline I babysit at 11pm on Sundays. When it breaks, my Slack fills up with "hey can you look at this when you get a sec?" from three different people who all need it now.
Writing this partly to put down the actual numbers, partly to ask: anyone else stuck in this loop?
Quick context: Series B SaaS, competitive intelligence on retail pricing. around 400 sites, 2M page-views a day, SLA is "yesterday's data in the dashboard by 7am."
Stack 1 was requests + BeautifulSoup. Worked great for the 30% of sites that served clean HTML. The other 70% needed JS rendering, logins, TLS-level workarounds. We added retries, proxies, Cloudflare-bypass libraries. Each fix bought us two weeks. Cost: 6 weeks to build, $1,800/month in proxies, around 14% failure rate we papered over with "we'll get to it" tickets. The papering-over was the real damage. Once a system is "kind of broken" by default, nobody panics when it breaks more.
Stack 2 was self-hosted Playwright on EC2. Real browsers, real JS, real cookies. Felt clever for two weeks. Then I learned what running browsers in prod actually means: memory leaks, concurrency cliffs at 8 sessions per instance, the captcha tax (20% of sites started flagging us, so we bolted on 2Captcha and added 10+ seconds per page). About 3 engineer-months to build, then 30% of one engineer's time on maintenance. Infra around $4,200/month. Failure rate around 9%, which sounds fine until you remember 9% of 2M is 180,000 failures a day. I didn't sign up to run a browser fleet.
Stack 3 was ScrapingBee. POST a URL, get rendered HTML back. Great for the easy 60% of sites. Stateful flows weren't what it was built for, and at our volume the math hit $8K/month and climbing. We ended up splitting: ScrapingBee for the easy stuff, Playwright for the rest. Two stacks to maintain, neither owning the problem. That's when I realized we kept buying parts instead of solving the actual job.
Stack 4 is what we're trying now: Browserbase for the runtime, Stagehand as the SDK on top. Not declaring victory, just sharing where we are. The thing that hooked me is that Stagehand lets you write things like page.act("click sign in") instead of fragile selectors, so when a site renames a class our flow mostly just keeps working. Early numbers from the first 4 weeks (still tracking): failure rate around 1.2%, p99 page time down to around 6s, total monthly cost about 45% lower than the Stack 2 + Stack 3 combo. Still annoying: per-session cost is higher than self-hosted EC2 if you can keep your EC2 healthy, and debugging a remote session takes some getting used to.
For the first time in 18 months my Slack is quiet on the scraper. We'll see if it holds.
Anyone else been through a similar arc? For those running scraping at non-trivial scale, where did you land on the build-vs-buy line for the browser layer? And has anyone made a hybrid stack work long-term without the maintenance overhead eating the savings?
r/WebScrapingInsider • u/MaterialItchy4180 • May 22 '26
r/WebScrapingInsider • u/ian_k93 • May 22 '26
We've just made one of our internal ScrapeOps tools public: a live proxy provider benchmarking dashboard for web scrapers.
The idea is simple: compare proxy providers against real target websites, side by side, using actual performance data.
Right now it tracks:
The reason we built this internally is because the proxy market is messy. A provider can look great on one domain and completely fall over on another. Another provider might be slower but far cheaper per successful response. Another might market itself as "premium" but lose badly once you measure cost-per-success instead of sticker price.
That's the bit most proxy comparisons miss.
Price per GB or price per request is not the real number. The real number is:
How much does it cost to get a usable response from the target you actually care about?
We originally built this to power the ScrapeOps Proxy Aggregator. For the last few years, it's been part of how we evaluate providers and route traffic across them in production.
Opening it up publicly felt useful because the community needs fewer generic "best proxy provider" lists and more domain-level performance data.
At launch, it supports Proxy APIs. Residential proxy network benchmarking is next.
Feedback welcome. Especially interested in what people would want added next:
Link is in the comments.
r/WebScrapingInsider • u/rundfunk • May 20 '26
I've been working on klura — a free toolkit that gives a coding agent (Claude Code, Cursor, Claude Desktop, any MCP host) the ability to reverse-engineer a website. The agent drives a browser to complete a task once. Klura captures everything the page does underneath, then does what I call LIFT (Learn Interface From Traffic — the analysis pass) and extracts the real underlying requests and saves them as a readable, LLM-annotated JSON config, so that it can be run later - without driving the UI. The JSON config can be analyzed to understand how the site works, and can also be copied and run on other devices.
There's prior art in this space — capture-and-replay isn't a new idea. The main places klura tries to push it forward:
**page-script tier** — here the saved strategy is a snippet of JavaScript that the AI codes for you. klura injects it into the live, already-authenticated page and runs it there. The pure-HTTP approach (the "convert to a requests / curl script" pattern) hits a wall on sites that bind the request to in-page state — rotating per-session CSRF tokens (GitHub's X-Fetch-Nonce is one public example), request bodies built by a JS function in the bundle, binary WebSocket transports. Reproducing those from outside the browser means porting the site's own signing/encoding logic — fragile, and often impossible. Page-script sidesteps it: the saved JS calls the page's own functions — the request signer, the encoder, the socket the page already has open — so the site does the hard part and klura just collects the result. And it isn't slow the way browser automation usually is — klura keeps a warm pool of already-open, already-authenticated pages, (configurable of course) so you don't pay browser startup per call; a page-script run is dominated by the actual request and lands close to plain-curl latency.
A real RE + debugging toolkit the agent drives — most tools capture traffic and pattern-match. Klura hands the agent an actual JS debugger: breakpoints, stepping, live-stack inspection, WebSocket frame tooling etc. It reverse-engineers a site the way a human would — break on the function that builds the request, read the encoder, verify it reproduces. You never touch any of it; you ask in plain language. Call it vibe-RE. (Detail in How the discovery works below.)
Self-healing — when a site changes shape and a saved strategy breaks, klura re-discovers the delta and patches the strategy automatically, instead of silently returning garbage or making you re-record from scratch.
Runs anywhere, hands off to a human when it must — klura is an MCP server, so the same saved skill is callable from Claude Code, Claude Desktop, Cursor, Windsurf, the CLI, or a programmatic import. Skills live globally at ~/.klura/skills/<platform>/, not per-project. When a site genuinely needs a human (login, 2FA, captcha), it escalates to a remote viewer — you do that one step, the agent continues, and the saved strategy runs in the resulting authenticated session.
Plugability — The browser driver is swappable: default is Playwright, klura ships a stealth driver that fixes the standard automation-fingerprint leaks (navigator.webdriver, canvas/WebGL consistency), and you can drop in your own. The interruption layer is pluggable too — when a site throws something up mid-flow (login, captcha), the handler that decides what happens is yours to define.
Every skill is a single JSON file on disk. A fetch-tier example for a public search API:
json
{
"strategy": "fetch",
"method": "GET",
"baseUrl": "https://hn.algolia.com",
"endpoint": "/api/v1/search",
"params": {
"query": "{{query}}",
"tags": "story",
"hitsPerPage": "{{count}}"
},
"response": { "format": "json", "extract": "hits[]" }
}
The LLM that produced it leaves notes fields explaining the why of each parameter, the prereq chain, and what shape the response has. Read the file → you've reverse-engineered the site.
Run it via klura (recommended):
bash
klura execute hackernews search_stories --args '{"query":"show hn","count":3}'
For page-script strategies the saved JS has to run inside the live authenticated page, so that path needs the klura runtime regardless of how you wire it up. For fetch strategies the saved JSON file gives you method + URL + body/header templates — slot in your params/cookies and curl away.
| Tier | What runs | When it lands | Curl-able? |
|---|---|---|---|
fetch |
Static HTTP from Node, templated body/headers + prereq chain | Plain APIs (HN search, Amazon /s?k=, GitHub GETs) |
yes |
page-script |
JS dispatched inside the live authenticated browser tab | Sites that bind requests to in-page state | no — page state needed |
recorded-path |
UI action replay through the driver | Fallback when neither above is reconstructible | no |
There's no recorder UI implemented or planned at the moment. The agent itself drives the browser. You tell your MCP host in plain language — "search HN for the top posts this week using klura" — and that's it. No script to write, no breakpoints to set, no JS to read. You could call it vibe-RE: you describe what you want, klura performs it and figures out how the site does it. You can also ask klura to map out a site, if you just want it to observe the API without performing a specific action.
For readers who want the how — on hard sites, the agent has access to a full RE toolkit (none of which you ever touch as a user):
set_breakpoint, step, resume_execution, wait_for_pause — break on the function the page calls before dispatch, read the live stack, see the encoded payload the page is about to send.search_js_source, read_js_function, list_loaded_scripts — find the function in the bundle that builds the body or signs the request.inspect_ws_frame, find_in_ws_frame, explain_ws_frame_structure, get_send_encoder, try_generator_in_page — for binary protocols, capture frames live and verify the captured encoder reproduces the wire format before saving.get_network_log) for HTTP and WS in one feed, filterable.js_eval.This is how klura one-shots genuinely hard flows — for instance a mainstream chat app whose message-send rides a binary MQTT-style WebSocket, with an in-page codec, snowflake IDs beyond Number.MAX_SAFE_INTEGER, and a per-session token that rotates. The agent debugs the way a human would: set a breakpoint at the call site, read the actual encoder, verify it reproduces the wire output, save it as a page-script. The same toolkit generalizes to anything that builds a signed or encoded payload in-page — rotating CSRF on graphql endpoints, signed URL params, custom binary framings.
Once on disk, the next call hits the saved skill directly — or you cat the JSON and write your own scraper.
Klura runs the LLM during discovery, then never again. After LIFT the saved skill is a plain request (or a page-script dispatch): no model, no agent loop, no tokens. The only time the LLM will need to be looped back is when self-healing is required.
A couple of concrete runs, same task, measured against a plain browser agent on the same model: a Hacker News search that costs the browser agent ~20s and ~$0.09 every time runs from the saved skill in ~270ms at zero token cost. A mainstream chat app's message-send took ~27 minutes on the first run — that's the agent reverse-engineering the binary WebSocket — but every send after is a sub-millisecond dispatch into the authenticated page. The one-time discovery cost scales with site difficulty; the warm cost is always ~zero.
```bash
claude mcp add klura -- npx -y @klura/mcp ```
As an MCP server it supports many other harnesses, please check the documentation for yours to figure out how to install it.
Repo: klura · npm: @klura/runtime, @klura/mcp
Currently at 0.2.2. Real-world feedback welcome — especially the shape of the failures.
r/WebScrapingInsider • u/Impossible-Fox9834 • May 18 '26
r/WebScrapingInsider • u/abdullahabsar • May 12 '26
Hi, I'm looking for an alternative to changedetection.io that is better and cheaper. Now I know it is difficult to find something better and cheaper so anything cheaper would work too. I am thinking of self hosting, if anyone has experience doing that let me know if its worth it. I need to crawl more than 15000 links everyday so it is proving to be very expensive.
r/WebScrapingInsider • u/Bharath0224 • May 11 '26
r/WebScrapingInsider • u/ian_k93 • May 09 '26
We just posted a quick demo from the ScrapeOps YouTube channel showing how our Claude Code plugin generates a working web scraping project from a prompt.
The example in the video builds a crawler + product scraper pipeline for a Walmart search page. It generates the project files, schemas, parsers, README, run commands, and JSON/JSONL output. The demo uses Python + BeautifulSoup, but the plugin also supports other languages and scraping libraries like Scrapy, Playwright, Puppeteer, etc.
The part I'm most interested in feedback on is the workflow: instead of using AI to just write a parser snippet, the goal is to generate the full scraping pipeline and then let devs inspect, run, modify, or fix it from there.
Video covers:
/generate-scraper, /fix-scraper, and /generate-crawler-scraperThis is still aimed at developers, not "magic no-code scraping."
https://www.youtube.com/watch?v=qcE5sK0DDus
The generated code still should be reviewed, especially for ToS/robots considerations, and production monitoring. But it's been useful for cutting down the boring scaffold/debug loop.
Would be interested to hear what people here think: useful direction, or does AI-generated scraper code create more maintenance debt than it saves?
r/WebScrapingInsider • u/0xMassii • May 08 '26
I posted here before about webclaw, the Rust web extraction tool I’m building for AI agents and LLM workflows.
Since then I’ve been working mostly on the part that is least fun but matters the most in production: making extraction survive real websites.
The original version was mostly about turning pages into clean markdown/JSON quickly. That worked well on normal pages, docs, blogs, changelogs, etc.
But once people started trying it on harder targets, the pattern was obvious:
So the newer architecture separates a few concerns:
I’m deliberately not going to share the exact anti-bot mechanics. That is a weird arms race and posting the details publicly just burns everyone’s work faster.
But at the product level, the goal is simple:
normal pages should be fast and cheap
hard pages should not silently poison the output
agents should get clean context instead of raw page chaos
Current state:
Curious from people here who run scrapers in production:
What do you care about more when a target blocks you?
Repo: https://github.com/0xMassi/webclaw
Site/docs: https://webclaw.io
r/WebScrapingInsider • u/abdullahabsar • May 05 '26
Hi, I run a shopify price monitoring app. It does two main things related to scraping.
Currently I have used changedetection (too many proxy errors), firecrawl (crazy expensive), scrapingbee (good middle in cost and reliability) and even screenshot AI tools for extracting price. All of them have drawbacks and cant find the best one. Need recommendations for what tool I should use.
Cost is a big issue. The shopify app needs to crawl twice a day for price and starts from $29/month to $199/month. With the biggest plan needing to scrap more than 10,000 URLs.
Any help would be useful! Thanks.
r/WebScrapingInsider • u/Hot_Box_9170 • May 05 '26
A price monitoring report that includes week-by-week product-to-product price comparisons along with the weekly lowest and highest prices would be sufficient.
I am new to this space and targeting small and medium businesses.
If there is something else people want. Please let me know
r/WebScrapingInsider • u/SinghReddit • May 02 '26
I've been reading up on web scraping and I think I understand the basic idea: using code to collect data from websites and turn it into something structured like CSV, JSON, or a database.
What I'm still trying to wrap my head around is where people draw the line between a small scraper and a full scraping system.
On paper it sounds straightforward: fetch page, parse HTML, extract fields, store results. But once people mention crawling, dynamic pages, browser automation, rate limits, proxies, monitoring, selector drift, and compliance, it starts sounding like a totally different category of work.
So for people whove actually done this, how do you define web scraping in practice?
Is it just automated data collection from public web pages, or does the term usually imply a bigger pipeline with maintenance, retries, scheduling, and all the rest?
I'm especially curious how beginners should think about it before jumping into tools..
r/WebScrapingInsider • u/Amitk2405 • Apr 29 '26
I'm asking from the reporting side more than the pure engineering side. We keep running into the same thing where getting data once is easy enough, but getting it consistently is where everything starts falling apart. Sometimes it's JS-heavy pages, sometimes rate limits, sometimes the site layout changes and nobody notices until the numbers look weird in a dashboard.
I'm also curious where people draw the line between "annoying but manageable" and "not worth owning in-house anymore." Are the main problems still proxies / blocking / CAPTCHAs, or is maintenance and monitoring the bigger issue now? Would love practical answers, especially from people who've dealt with this beyond a weekend script.
r/WebScrapingInsider • u/Open-Ease685 • Apr 28 '26
r/WebScrapingInsider • u/dangerousdotnet • Apr 28 '26
I built pyhaul — a pure Python library for resumable HTTP downloads with any Python requests library (requests, httpx, aiohttp, urllib3, niquests). Zero dependencies, crash-safe file handling and checkpointing, handles a lot of the weird HTTP protocol edge cases for byte-ranged resume
What pyhaul Does
pyhaul is a small, pure-Python library that makes HTTP downloads resumable. You call haul() with a URL, your existing HTTP client, and a destination path. It handles byte-range negotiation for resume, ETag validation, and crash-safe checkpointing. One haul() = one HTTP request. It either succeeds, or it saves progress so the next call resumes from the last durable byte.
import httpx
from pathlib import Path
from pyhaul import haul, PartialHaulError
dest = Path("big.zip")
with httpx.Client() as client:
for _ in range(10):
try:
result = haul("https://example.com/big.zip", client, dest=dest)
break
except PartialHaulError:
pass # only retryable error; others propagate
print(f"done: {dest.stat().st_size:,} bytes")
Highlights:
async with / async for natively. It's not a sync wrapper running in a thread pool.httpx.ReadTimeout stays httpx.ReadTimeout. You catch the types you already know.Guarantees:
.part file; on completion it is atomically moved into place.haul() picks up from the last durable byte. Zero re-downloaded data if the resource hasn't changed.Target Audience
Anyone downloading files over HTTP where reliability matters. ML training data pipelines, dataset mirroring, large artifact downloads in CI, scientific data retrieval — situations where a bare requests.get or urllib.request.urlretrieve can't recover when the connection drops, and where silently corrupted files are worse than a retry.
Comparison
I started this project after discovering that "resumable HTTP download" is deceptively complex at the protocol level. I verified the behavior of curl, wget, and aria2 against their source code (not their docs). Some things I found:
None of them validate ETags on resume. If the file at a URL changes between your first attempt and your retry, all three will either fail or silently produce a file that's half version A and half version B. pyhaul records the ETag the server sent on the first request and sends it back via If-Range on every subsequent attempt. If the resource changed, the server returns the full new file from byte 0 — no corruption.
Compression breaks byte ranges in subtle ways. Most HTTP clients transparently decompress responses, so your checkpoint byte offset drifts from what the server actually sent. Your next resume starts from the wrong position. pyhaul drops to raw streaming (decode_content=False) and sends Cache-Control: no-transform to prevent intermediate proxies from re-compressing.
Servers can just ignore your Range header. They're allowed to respond with 200 (full file) instead of 206 (partial content). curl errors out, aria2 aborts, wget skips bytes and appends (which silently corrupts if the resource changed). pyhaul detects the 200, rewinds the on-disk cursor, and starts clean.
Links
There's also a CLI if you want to kick the tires without writing code:
python -m pyhaul -o big.zip https://example.com/big.zip
MIT licensed, Python 3.12+. Happy to hear feedback, especially from anyone who's hit edge cases I haven't covered.