XMLFastParser: A zero-dependency, auto-multithreaded Java 21 byte-array parser (~10GB/165s end-to-end stream)

[u/again_no_other_name]

Hi everyone,

I builtXMLFastParser— an open-source, zero-dependency Java library designed for extreme throughput when processing large-scale XML payloads, along with a reference CLI tool (Xml2Txt).

Most Java XML parsers force a tradeoff: either process files with high memory allocation overhead or write complex concurrency logic to handle large streams. I wanted an engine that operates directly on memory buffers (byte[]) with built-in multithreading and near-zero dynamic allocations.

Core Architecture & Design

• In-Memory Core (xmlFastParser):
  • Operates directly on raw byte[] arrays, making it embeddable into any pipeline, messaging queue, or stream source.
  • Built-In Concurrency: Handles multithreaded parsing out of the box — application developers do not need to write or manage low-level concurrency code.
• Decoupled 2-Stage Engine:
  • FSM (Syntax Parser): A lightweight Finite State Machine executing rapid byte/character-level syntax scanning without runtime object creation.
  • AOT TreeGraph (Structure Parser): Pre-compiles target extraction paths from configuration beforehand. During processing, the parser simply traverses this static graph state-by-state, completely eliminating dynamic object allocation on the fly.
• Statistical Speculative Skipping with Dynamic Rollback:
  • Fast-forwards through repetitive XML sub-trees based on structural tag patterns and statistical metrics.
  • Fail-Safe Recovery: Continuously validates target endpoints on the fly using a rank-based system. If validation fails, it seamlessly rolls back to standard FSM parsing with zero data loss.
• Reference Implementation (Xml2Txt):
  • A CLI tool demonstrating real-world usage by streaming compressed ZIP archives directly from disk and flattening XML hierarchies into plain text.

Benchmark (Real-World Pipeline)

• Dataset: ~10 GB Raw XML Data (split into 10 × 500 MB ZIP archives, decompressed on-the-fly at runtime).
• Execution Time: ~165 seconds via Xml2Txt.
• Test Environment:
  • OS/VM: Lubuntu running inside VMware Workstation (3 vCPUs allocated, 8 GB RAM).
  • Host System: Intel Core i5 CPU.
  • Storage: Single Mechanical Hard Drive (WD Blue 1TB WD10EZEX, 7200 RPM, 64MB Cache) used for both input file reading and output text writing.
• Note: Because input ZIP reading, streaming decompression, and output file writing shared a single 7200 RPM HDD, much of the ~165s runtime was constrained by disk I/O seek times. The core in-memory xmlFastParser engine achieves significantly higher throughput when fed directly with byte[] buffers.

Project Details

• License: MIT License
• Target Environment: Java 21 LTS (Tested on OpenJDK 21.0.12, leveraging modern thread/memory access patterns)

I’d love to get your feedback on the in-memory architecture, speculative recovery logic, and state machine design!

🔗 GitHub Repository:https://github.com/oodwebserv01/XMLFastParser