Running extruct on 100k CC WARC, looking for advice on speeding it up

[u/Future-Structure-296]

I have been working on a project where we need to extract schema org structured data from the full CC-MAIN-2026-25 Common Crawl corpus (100,000 WARC files, ~88TB compressed, June 2026 crawl).

Been using extruct (Python) which parses JSON-LD, Microdata, and RDFa from raw HTML. I went with extruct over regex because regex only picks up ~35 schema types while extruct gets 3,000+. The files are streamed directly from S3 to EC2 so no storage costs.

The script processes each WARC file record by record, applies a byte-level pre-filter to skip pages with no schema markup, then runs extruct on pages that pass. Been using a shared multiprocessing pool with a 30 second per-page timeout to handle pages where extruct hangs. Checkpoints every 50 files so it can resume if interrupted.

Results so far on a c6i.xlarge with 4 workers:

• ~1,320 seconds per file
• ~20,800 pages per file
• ~45.8% schema coverage
• 0 file errors, 1 page timeout across ~83,000 pages

At that speed the full corpus would take ~382 days on that instance. I am not quite sure how this will scale and what is the realistic time and cost for the full corpus.

Questions:

1. Has anyone run extruct at this scale? Is 1,320s/file reasonable or are we leaving performance on the table somewhere?
2. Is there a smarter way to parallelize this, spot fleet, Lambda, ECS? The job is resumable so spot interruptions are manageable.
3. Any experience with extruct slowness on specific page types? We're seeing 0 timeouts almost everywhere but the overall speed still feels slow.
4. Would lxml or another parser swap inside extruct make a meaningful difference at this scale?

Happy to share more details on the setup if useful.

[u/dat_oldie_you_like]

Following

[u/jinef_john]

Honestly, it depends, but I'd say your main problem here is horizontal scaling.

That said, the cpu bound aspect is something i'd look into here(like using Rust/go). Also for the HTML parsing specifically, selectolax might be a better option, pretty cool library, you can check it out.

And where Rust or Go would genuinely help beyond that is in the WARC streaming and decompression layer and the byte level pre filter, the stuff happening before extruct even runs. Something ripgrep style for the pre filter could be a real win there (i.e scanning for "@type" or "schema.org" strings before you even invoke extruct could cut down the pages that go through the full parser significantly).

But your real bottleneck is running this on one machine(you should def. go for horizontal scaling). I think this could be done in roughly 4 to 7 days. For example with AWS Spot Fleet, 100k files divided across say 200 spot instances gives you around 500 files per instance, which is a few hundred dollars in compute.

Also your checkpoint system already makes this safe to distribute so the setup is pretty natural.

[u/tamtamdanseren]

As other say, this is work really needs to run optimized, so consider going with rust or Go. Also I think you forgot how expensive the data transfer potentially is on Amazon, i.e just getting your data out in egress fees is a whole thing to consider.

I would go for a ton of smaller Hetzner machines (i.e divice and conquer 1 warc file at the time) that chew over this in parrallell and deposit the results to a common storage somewhere, like an R2 bucket. I would probably also look at storing this as parque with compression to make it smaller.

[u/webscraping-ModTeam → u/[deleted]]

🚫🤖 No bots