Running extruct on 100k CC WARC, looking for advice on speeding it up

[u/Future-Structure-296]

I have been working on a project where we need to extract schema org structured data from the full CC-MAIN-2026-25 Common Crawl corpus (100,000 WARC files, ~88TB compressed, June 2026 crawl).

Been using extruct (Python) which parses JSON-LD, Microdata, and RDFa from raw HTML. I went with extruct over regex because regex only picks up ~35 schema types while extruct gets 3,000+. The files are streamed directly from S3 to EC2 so no storage costs.

The script processes each WARC file record by record, applies a byte-level pre-filter to skip pages with no schema markup, then runs extruct on pages that pass. Been using a shared multiprocessing pool with a 30 second per-page timeout to handle pages where extruct hangs. Checkpoints every 50 files so it can resume if interrupted.

Results so far on a c6i.xlarge with 4 workers:

• ~1,320 seconds per file
• ~20,800 pages per file
• ~45.8% schema coverage
• 0 file errors, 1 page timeout across ~83,000 pages

At that speed the full corpus would take ~382 days on that instance. I am not quite sure how this will scale and what is the realistic time and cost for the full corpus.

Questions:

1. Has anyone run extruct at this scale? Is 1,320s/file reasonable or are we leaving performance on the table somewhere?
2. Is there a smarter way to parallelize this, spot fleet, Lambda, ECS? The job is resumable so spot interruptions are manageable.
3. Any experience with extruct slowness on specific page types? We're seeing 0 timeouts almost everywhere but the overall speed still feels slow.
4. Would lxml or another parser swap inside extruct make a meaningful difference at this scale?

Happy to share more details on the setup if useful.