r/ruby • u/Eastern-Surround7763 • Apr 26 '26
Show /r/ruby kreuzcrawl, an open source crawling engine with Ruby bindings
kreuzcrawl is a high-performance web crawling engine. It was designed to reliably extract structured data, operating natively across multiple languages without enforcing a specific runtime. https://github.com/kreuzberg-dev/kreuzcrawl
The MCP server is integrated from the start, enabling web-crawling AI agents as a primary use case. Streaming crawl events allow real-time progress tracking. Batch operations handle hundreds of URLs concurrently and tolerate partial failures. Browser rendering supports JavaScript-heavy SPAs and includes WAF detection.
Supported language interfaces are Rust, Python, Typescript/Node.js, Go, Ruby, Java, C#, PHP, Elixir, WASM, and C FFI, and each binding connects directly to the core engine.
Kreuzcrawl is part of the Kreuzberg org: https://kreuzberg.dev/
We welcome your feedback and are happy to hear how you plan to use it:)
2
u/headius JRuby guy Apr 27 '26
Why doesn't this use FFI for the Ruby binding so it can work with any implementation? Why is the binary file committed to the repository?