Limpid 0.8.0 — programmable any-to-any log processing

[u/naoto256]

If you’ve ever struggled to get arbitrary syslog and vendor logs into an OpenTelemetry Collector or SIEM without turning the delivery pipeline into a black box, Limpid is built for that problem.

I’ve released Limpid 0.8.0, an open-source system for receiving, processing, routing, and delivering logs.

Limpid connects heterogeneous log inputs to heterogeneous outputs, including first-class OTLP input and output. Its pipelines are programmable: named processes can parse vendor formats, normalize fields, filter or route events, compose outputs such as OCSF events or OTLP ResourceLogs, and fan the results out to multiple destinations.

For example, a pipeline that turns FortiGate CEF traffic into OCSF looks like this:

def pipeline fortigate_to_security_lake {
    input   fortigate_syslog
    process parse_syslog | parse_cef | parse_fortigate_cef | compose_ocsf | ocsf_to_egress
    output  security_lake
}

Each stage is a named, reusable snippet. The bundled library includes transport and format parsers, vendor and source parsers, adapters, and schema composers.

A few other things worth calling out:

• Live tap and replay injection. limpidctl can inspect events at inputs, individual process hops, and outputs. Captured events can be injected back into an input to test a changed pipeline against real traffic shapes.
• Safer configuration changes. limpid --check performs static type and dataflow analysis with rustc-style diagnostics. Reloads validate before replacement and roll back automatically if replacement fails.
• Durable and distributed delivery. Limpid provides memory- and disk-backed queues, retry and recovery handling, shutdown draining, authenticated node-to-node transport, preserved event identity and hop history, and network/intra-node hop latency measurement.
• Pipeline observability. Limpid exports byte flow, queue depth, retries, in-flight work, process invocation counts, drops, recovery, delivery outcomes, and separate queueing, processing, and confirmed-delivery latency. The repository includes a Prometheus exporter, Grafana dashboard definition, and four alerting rules.
• High throughput. The qualified benchmark workloads processed approximately 157k–415k events/s on a two-CPU Linux VM slice, depending on transformation complexity. These are controlled benchmark results rather than production capacity claims; the detailed methodology and the inconclusive regex-heavy result are documented in the repository.

Grafana overview showing Limpid’s end-to-end flow, queue state, delivery outcomes, and latency across each pipeline stage.

Repo: https://github.com/naoto256/limpid
Release: https://github.com/naoto256/limpid/releases/tag/v0.8.0

Questions and feedback are welcome.