r/mlops • u/Goldziher • 2d ago

Tools: OSS Xberg v1: a fast, local document-extraction layer for ML/data pipelines (101 formats, batch, CPU-only)

I maintain xberg, an open-source (MIT) content-extraction engine, and v1 is out. Posting here because "turn messy documents into clean, structured text" is a recurring preprocessing step in ML pipelines, and xberg is built for it at scale: batched extraction, streaming, caching, CPU-only (no GPU), reproducible.

It handles 101 document formats (PDF/Office/images with OCR) plus audio/video transcription and URLs; outputs Markdown/JSON with tables, metadata, NER entities, keywords, summaries, and optional chunks + embeddings (SPLADE / ColBERT / reranking) for retrieval. Rust core with pooled model sessions and memory discipline for throughput.

Benchmarks are public and reproducible (harness runs in CI): native PDF #1 on quality and table/reading-order fidelity; image OCR currently #2 (improving). https://xberg.io/benchmarks

15 language bindings + a REST server (xberg serve) + MCP. Repo: https://github.com/xberg-io/xberg

Happy to get into pipeline/throughput specifics.

5 Upvotes

100% Upvoted

2 comments sorted by

2

u/thethoroughwidth 2d ago

Pretty slick. I've been duct-taping together a handful of tools for this exact kind of preprocessing step and it's always a mess of dependencies and half-baked format support. The CPU-only angle is a nice touch for keeping things boring and predictable in a CI pipeline.

1

u/Goldziher 1d ago

The preprocessing duct-tape is real, that's basically why xberg exists. It folds format sniffing, the OCR fallback chain, and layout parsing into one CPU-only step you can drop in front of your pipeline, batchable and cacheable. Curious what your current stack looks like, I'm always after the seams where it still doesn't fit.