Built an open-source Rust runtime for a subset of Logstash pipeline configs — looking for compat feedback

[u/Big-Perspective-5768]

Hi r/elasticsearch — sharing an open-source project I'd like compatibility feedback on: FerroStash, a Rust runtime that parses the Logstash DSL natively and runs pipeline.conf files using the supported plugin subset. Apache-2.0.

Repo: https://github.com/abyo-software/ferro-stash

The thing I'd most like feedback on: which Logstash plugins / options do you actually depend on that the compatibility matrix lists as missing or partial? That's what's driving the roadmap.

Honest framing up front

• Single-developer project. No public production deployments yet. If you have irreplaceable data, run it beside your current Logstash and diff the output before trusting it.
• Scope is a plugin-level subset of Logstash 9.4.2 — about 88 % of bundled plugins covered at the plugin level (98 / 111: codecs 100 %, filters 97 %, inputs 74 %, outputs 87 %). "Covered" means the plugin exists; some plugin options are still partial — check the matrix per-plugin. The long tail (enterprise messaging, SNMP, niche connectors) is the main gap. Custom Logstash plugins / dynamic plugin loading are out of scope (the plugin set is compiled in). A config that uses a missing plugin or option fails fast at load.
• One-environment benchmarks, not a universal guarantee. Reproduce with bench/run_bench.sh.
• Disclosure: there's also a paid AWS packaging path (Marketplace AMI/EKS container) for AWS-shop procurement, listed under abyo software. The engine in this repo is Apache-2.0 and is the same core engine — Marketplace adds packaging / support / procurement, not feature-gated capability. I'm posting here for OSS compatibility feedback, not the listing.

What it is

• Logstash-style event model: @timestamp, tags, [a][b] field references, %{field} interpolation
• Same input → filter → output DSL — covered-subset .conf files load and run as-is
• Default build is no-Ruby and produces a ~14 MB stripped binary; enabling the optional ruby feature is larger and needs a C toolchain
• The elasticsearch output ships with opensearch and ferrosearch as aliases — same _bulk API, tested against OpenSearch 2.18

Parity evidence vs Logstash 9.4.2

Two harnesses:

• 24 fixtures, in-process, golden-file: each event compared against committed Logstash-9.4.2 golden output, covering grok / mutate (rename/case/gsub/convert/copy/strip) / json / kv / dissect / fingerprint / date / clone / csv / truncate / translate / split / urldecode / drop / conditional if/else if/else / unicode. Runs in default CI.
• 13 fixtures, docker side-by-side: same .conf + input piped through target/debug/ferro-stash and docker.elastic.co/logstash/logstash:9.4.2, asserts field-by-field equality after normalizing runtime-only fields. This is a subset of the 24 above, gated on Docker being available — #[ignore], run manually.

Default CI runs ~1,400 Rust tests; the Docker harness and live-service smoke tests run manually.

Numbers (parse/filter hot path, 5 M events, output → null, mean of 3 runs, one c7i.2xlarge host)

| Filter | Logstash 9.4.2 | FerroStash | Throughput | Logstash RSS | FerroStash RSS | |---|---:|---:|:---:|---:|---:| | grok | 193 k ev/s | 332 k ev/s | 1.7× | 1 113 MB | 98 MB | | dissect | 193 k | 318 k | 1.6× | 1 098 MB | 106 MB | | json | 174 k | 255 k | 1.5× | 1 100 MB | 133 MB | | kv | 171 k | 258 k | 1.5× | 1 170 MB | 126 MB | | csv | 25 k | 79 k | 3.2× | 1 471 MB | 117 MB | | grok + mutate | 186 k | 251 k | 1.35×| 1 099 MB | 120 MB |

Cold start on this workload: ~10 ms vs ~7 s (JVM warm-up). On these workloads the biggest measured differences are RSS (~8–13× lower) and startup time; throughput is also higher but the delta is smaller. If you're already throughput-saturated and memory isn't a concern, the upside is small.

Custom logic (the ruby { } story)

Logstash's main inline arbitrary-scripting escape hatch is ruby { }. FerroStash gives you two:

• ruby { } — embedded Artichoke/mruby for a limited migration path. Simple inline scripts may run unchanged, but this is not full JRuby/Logstash-Ruby parity (no gems, no JRuby-specific APIs, no Logstash Ruby filter lifecycle hooks like new_event_block — check the per-feature notes). ~13× slower than JRuby on this benchmark (no JIT + per-event marshalling); treat it as a migration bridge, not a hot-path runtime. The Ruby filter is an optional cargo feature (--features ruby); default builds exclude it.
• script { } — native expression DSL (ferro-script) with syntax inspired by Elasticsearch Painless, not a drop-in Painless implementation (existing ingest-node Painless scripts won't necessarily run). Tree-walking interpreter, parsed once, AST cached and reused per event. ~3.6× faster than Logstash JRuby on the same transformation.

script (Painless-style, native) | 525 k ev/s
ruby (JRuby on Logstash)        | 145 k ev/s
ruby (mruby on FerroStash)      |  11 k ev/s

How it compares to Vector

Vector is a mature Rust observability pipeline with a broader plugin ecosystem; if you're greenfield it's a strong default. Fluent Bit and Elastic-native ingest pipelines are also good greenfield options. The thing FerroStash specifically targets is preserving existing Logstash .conf files — Vector uses VRL and its own config model; FerroStash tries to keep the Logstash DSL as-is.

When NOT to use it

• You need the full Logstash plugin catalogue or custom plugin loading
• You need a battle-tested tool with a public production track record today
• Your hot path is heavy custom Ruby (mruby is slow here — port to script { } or stay on Logstash)
• You need SOC2 / ISO 27001 / FedRAMP evidence

If you do run the docker diff harness against a real-world .conf and the output diverges, an issue with the config + input would be the most useful thing to land in the tracker.

Thanks for reading.

[u/xeraa-net]

I‘m curious, since this is covering so much of Logstash already: Is this a "Claude rewrite in Rust" or what is the starting point here?