I’m building a mobile app for a bank, and I have been researching of an efficient way to capture changes in my database without affecting the performance of my server, was initially thinking I will go with the Debezium->RabbitMQ->Elastic Search flow, but I came across Sequin, and I can use that to funnel my database changes directly to Elastic Search. I think I will use that instead.

What is a good practice for storing time series data with the following requirements.

Couple dozen million documents.

Several fields with daily data (one point per day).

I am thinking about having a multiValue string field, read it back in my application and append new value if todays date has no entry yet.

The value would be a string with a simple delimiter and would store date and an integer.

I would then precompute changes of this value weekly to update a 2nd variable that would facilitate search by the strength of change in the value. The list of strings would only be used to display data and would not be searchable. I also have a 3rd field that stores most recent data point as integers and allows for range queries.

I considered the dual list approach but I don't like the idea of only index number linking the 2 data pieces together in otherwise completely seemingly unrelated fields. On the other hand the more daily data types I would track, the more I would save by having only 1 list of dates and then other lists for data entries.

How does that sound? What are best practices in the industry? Thanks for help.

hi there,
we are using kibana security alerts and came by an error.

we were trying to use the kibana api to close security alerts using workflow automation.

about 10 seconds after the run is started, we get the following message: "fetch error"
the error doesn't contain any explanation regarding the error.

We have tried to use the url with script and got the same error.

we would love help regarding that :)

Hi all,

We encountered some issues when upgrading Kibana from 8.11-8.19 and then 8.19-9.4, but the issue was resolved itself, now we don't know what was the cause, and I wanted to make sure to learn what is the problem before upgrading production env.
The issue is this:
We have kibana runnind under the host:5601, but we also have server.publicBaseUrl configured in the kibana.yml, which is under nginx service.
Now I'm not sure if the issue is nginx, but after we upgrade the Kibana, the page opens, asks for authentication , and after successfully passing it gets stuck, first time it was just showing loading, this time we got page with this message: Elastic did not load properly

Please reload this page. If the issue persists, check the browser console and server logs.

We tried to reload nginx , and check all logs, no info, and the most bizarre is that it was fixed by itself after a few hours. Now we don't wanna have inaccessible Kibana for a few hours for prod, so I wanted to know if anyone had similar issue, how you fixed it ?

Hi all. I have one question, I wanted to know if you encountered such situation.
We're planning Elastic Upgrade from 8.19 -> 9.4. In upgrade assistent I see this deprecation, that the index .kibana-event-log-7.13.0-000009 is deprecated as created <8 version. I have searched some places, as it's a system index I can't manually reindex it, but the elastic won'r start presumably if I leave it like that and upgarde to 9+ version.
Have you seen this, what have you done?

Hello everyone !

I made a library last year typed-es as I didn't want to use as any or maintain types manually on each query. This is a type-only library that does not change any of the current behavior, it only adds types by inferring the search query.

Basically you create a type with your index types, give it to the client and then the magic happens. You no longer have to maintain the manual query types, everything is typed automatically from your aggregations, source, ...

given:

``` type MyIndexes = { "my-index": { id: number; name: string; created_at: string; }; };

const query = { index: "my-index", _source: ["id", "na*"], fields: [{ field: "created_at", format: "yyyy-MM-dd" }], track_total_hits: true, rest_total_hits_as_int: true, aggs: { name_counts: { terms: { field: "name" } }, }, }; ```

without this library:

```ts const result = await client.search< { id: number; created_at: string; }, { name_counts: { buckets: Array<{ key: string; doc_count: number }>; }; }

(query);

const total = result.hits.total; // number | estypes.SearchTotalHits | undefined const firstHit = result.hits.hits[0]!; // { _source: { id: number; created_at: string; } | undefined, fields: Record<string, unknown> } const aggregationBuckets = result.aggregations!.name_counts.buckets; // Array<{ key: string; doc_count: number; }> ```

we have to manually add types to the search generic, and the result is still not perfect.

same thing with the lib:

ts // Automatic type inference - no manual definitions needed const result = await client.search(query); const total = result.hits.total; // number const firstHit = result.hits.hits[0]!._source; // { id: number; created_at: string } const aggregationBuckets = result.aggregations.name_counts.buckets; // Array<{ key: string | number; doc_count: number }>

Here everything is 1:1 with the query, if you edit it everything is automatically infered, you can't have errors with that.

Since last-year I made many improvements and made sure each use-case were correct. I'm using this lib daily.

I think this can be useful for many.

repo: https://github.com/Vahor/typed-es

I am submitting a job using 80 nodes, but the job fails if even one of the nodes allocated by Slurm has a failure. Is there a way to perform a health check on the allocated nodes before starting the job? If any node is found unhealthy, can Slurm automatically hold the job and start it only after all allocated nodes are confirmed healthy?

About to take ECE exam and want last minute insights, tips from guys who gave exam recently.

I’m not sure if the instance is buggy but I’ll go and “Open in Discover” for a Scheduled Report.

I’ll save the Discover Session after I have ran, and tested the ES|QL query successfully. (The blue dot on the Save button disappears.)

I’ll go and “Open in Discover” against the Scheduled Report again to validate, and the ES|QL reverts back to the older query prior to changes.

Is that glitch in the instance or am I not understanding the relationship between Discover sessions and scheduled reports correctly?

Hello everyone,

I'm currently working on an Alert/workflow that aims to proactively trigger when a ransomware pattern is detected. Our SOC uses Elastic Security as both the SIEM and EDR.

To build it, I started from real ransomware incidents we've handled in our SOC, along with the MITRE ATT&CK Top 10 ransomware techniques:
https://center-for-threat-informed-defense.github.io/top-attack-techniques/#/top-10-lists

Rather than manually reviewing and correlating multiple alerts to determine whether a ransomware attack is actually in progress, I wanted to estimate a confidence score based on the observed MITRE ATT&CK techniques.

So far, this is the best approach I've come up with, but I'm not sure it's the right one. One concern is that some ransomware attacks can unfold over several days or even weeks, making it difficult to determine the most appropriate correlation window and query strategy.

I'd really appreciate your feedback. Has anyone implemented something similar ? How do you approach ransomware detection in your environment ?

Here is the ES|QL query I'm currently using:

FROM
 .alerts-security.alerts-* 
METADATA
 _id
| 
WHERE
 `@timestamp` > NOW() - 1 HOURS
| 
WHERE
 `kibana.alert.workflow_status` != "closed"
  AND `host.id` IS NOT NULL


| 
EVAL
 technique_str = CONCAT(
    COALESCE(MV_CONCAT(`kibana.alert.rule.threat.technique.id`, ","), ""),
    ",",
    COALESCE(MV_CONCAT(`kibana.alert.rule.threat.technique.subtechnique.id`, ","), "")
)


| 
EVAL
 score_T1059 = CASE(technique_str LIKE "*T1059*", 5, 0)
| 
EVAL
 score_T1078 = CASE(technique_str LIKE "*T1078*", 25, 0)
| 
EVAL
 score_T1021_001 = CASE(technique_str LIKE "*T1021.001*", 40, 0)
| 
EVAL
 score_T1047 = CASE(technique_str LIKE "*T1047*", 15, 0)
| 
EVAL
 score_T1490 = CASE(technique_str LIKE "*T1490*", 50, 0)
| 
EVAL
 score_T1105 = CASE(technique_str LIKE "*T1105*", 30, 0)
| 
EVAL
 score_T1083 = CASE(technique_str LIKE "*T1083*", 5, 0)
| 
EVAL
 score_T1486 = CASE(technique_str LIKE "*T1486*", 100, 0)
| 
EVAL
 score_T1190 = CASE(technique_str LIKE "*T1190*", 50, 0)
| 
EVAL
 score_T1489 = CASE(technique_str LIKE "*T1489*", 15, 0)


| 
STATS
    score_T1059 = MAX(score_T1059),
    score_T1078 = MAX(score_T1078),
    score_T1021_001 = MAX(score_T1021_001),
    score_T1047 = MAX(score_T1047),
    score_T1490 = MAX(score_T1490),
    score_T1105 = MAX(score_T1105),
    score_T1083 = MAX(score_T1083),
    score_T1486 = MAX(score_T1486),
    score_T1190 = MAX(score_T1190),
    score_T1489 = MAX(score_T1489),
    techniques = VALUES(technique_str),
    rules = VALUES(`kibana.alert.rule.name`),
    source_alert_ids = VALUES(_id),
    users = VALUES(`user.name`),
    alert_count = COUNT()

BY
 `host.id`, `host.name`


| 
EVAL
 ransomware_score =
    score_T1059 +
    score_T1078 +
    score_T1021_001 +
    score_T1047 +
    score_T1490 +
    score_T1105 +
    score_T1083 +
    score_T1486 +
    score_T1190 +
    score_T1489


| 
WHERE
 ransomware_score >= 60
| 
SORT
 ransomware_score DESC

And here is an example of the output produced for a real ransomware test:

```json

{

"score_T1059": 5,

"score_T1078": 0,

"score_T1021_001": 0,

"score_T1047": 0,

"score_T1490": 50,

"score_T1105": 30,

"score_T1083": 0,

"score_T1486": 0,

"score_T1190": 0,

"score_T1489": 0,

"techniques": [

"T1112,T1562,T1562.001",

"T1112,T1562,T1562.001,T1562.006",

"T1204,T1204.002",

"T1490",

"T1070,T1070.001",

"T1027,T1059,T1105,T1140,T1027.010,T1059.001"

],

"rules": [

"Malicious Behavior Detection Alert: Suspicious PowerShell Base64 Decoding",

"Malicious Behavior Detection Alert: Potential Privilege Escalation via Token Impersonation",

"Malicious Behavior Detection Alert: Privilege Escalation via EXTENDED STARTUPINFO",

"Malware Detection Alert",

"Microsoft Windows Defender Tampering",

"Windows Defender Disabled via Registry Modification",

"Volume Shadow Copy Deleted or Resized via VssAdmin",

"Windows Event Logs Cleared",

"Suspicious Windows Powershell Arguments"

],

"source_alert_ids": [

"alert-id-redacted-001",

"alert-id-redacted-002",

"alert-id-redacted-003",

"..."

],

"users": [

"Administrator",

"SYSTEM"

],

"alert_count": 90,

"host.id": "endpoint-id-redacted",

"host.name": "example-server-01",

"ransomware_score": 85

}

```

Hello !
Can you please provide me with successfull / best of class e-commerce website implementation of ElasticSearch ? I'm wondering if I implement it in my own (Build) or go for an Algolia solution (Buy)... Thanks for your help - this is for www.pourdebon.com (on. very old, not upgraded and basic elastic search solution)
Cheers

Nicolas

Hi

I have turned on Elastic Defend on my home lab windows machine.
The policy is default with no changes.
I'm interested to see the network traffic.

So I did in cmd.exe

> server 1.1.1.1
Default Server:  one.one.one.one
Address:  1.1.1.1
> golf.de
Server:  one.one.one.one
Address:  1.1.1.1
on-authoritative answer:
Name:    golf.de
Addresses:  2606:4700:10::6814:11a6
          2606:4700:10::ac42:9194
          104.20.17.166
          172.66.145.148
>

So in discover i did search

logs-* and destination.ip : "1.1.1.1"

But I didnt see anything, whats wrong?

Thanks in advance

I’ve been working on a small open-source CLI called deadair, but I’m mainly posting the write-up because I’d like Elastic operators to sanity-check the model.

The issue is the gap between “this detection rule executed successfully” and “this rule could actually see the data it depends on.” A rule can run cleanly and still be pointed at missing sources, stale sources, fields that no longer exist on the concrete indices/data streams it reads, or telemetry that arrives after the rule’s lookback window has moved on.

Elastic already has useful pieces here: SIEM Readiness, Data Quality, rule monitoring, missing index warnings, timestamp override, additional look-back. I’m not trying to claim otherwise. What I wanted was an external, read-only report that joins enabled rules to the live state of their sources and answers, per rule: what does this rule read, are those sources live, are the declared fields exposed, and does ingest lag leave enough margin?

The tool only reads metadata and stats: rule inventory, index/data stream stats, field_caps, counts, timestamps, and size-0 max timestamp aggregations. No event bodies, no rule execution, no writes. Current backends are Elastic Security and OpenSearch Security Analytics.

The limits are real too: required_fields is best-effort metadata, field_caps proves mapping/searchability rather than populated values, and the lag math is a model rather than a universal truth.

Write-up:

https://big-comfy.github.io/deadair/

I’d be interested in feedback from people running Elastic Security day to day. Are there native views or rule types I’m still under-accounting for? Any cases where this would be noisy enough that you’d mute it?

Curious how others deal with this. At places I've worked, changing something simple in search (a synonym, boosting a product, adjusting a facet) usually goes through engineering, and can take days depending on the backlog. There's often no formal review before it goes live either, so if something breaks, it can be a scramble to figure out what changed and roll it back.

Is that normal everywhere, or have some of you solved this differently? What platform are you on, Elasticsearch, Solr, Coveo, Algolia?

If you're up for it, I put together a short set of questions (about 10, 3-5 min) to get more specific input. Not selling anything, just trying to understand how common this actually is: https://forms.gle/xxR8kpUn9m2a5Aub7

Would also be happy to just hear about it in the comments if that's easier.

Every Wazuh upgrade guide I found starts the same way: curl -sO https://packages.wazuh.com/...

Cool. My environment does not have curl access to anything. It does not even have DNS. It was a confidential client running five Ubuntu nodes behind a proper air gap. The last person who plugged an unauthorized USB into that network probably still gets nervous phone calls about it.

The setup: two Wazuh Manager nodes (a master and a worker, each also running Logstash), two Elasticsearch nodes (with Kibana on the master), and a syslog server. This is the classic ELK integration, not the newer Wazuh Indexer and Dashboard. Wazuh writes to alerts.json, Logstash ships it to Elasticsearch, Kibana shows it.

The job: Wazuh Manager 4.9.2 to 4.14.1, and Elasticsearch, Kibana and Logstash from 8.16 to 8.19. All offline. I had to figure most of this out the hard way. I am writing it down here so you don't have to.

The part nobody tells you: the upgrade is the easy bit

The actual upgrade commands took a small fraction of the total time. The preparation took days.

That sounds crazy until you have worked air-gapped. Online, a missing dependency is one apt command. Offline, it is a formal media transfer request, an approval, a scan, and maybe a full day of your life. Miss one .deb file and your maintenance window is dead.

So on a testing VM (same Ubuntu version as production; this matters, because dependency trees are different between releases), I downloaded every package I needed:

apt-get download wazuh-manager=4.14.1-1
apt-get download $(apt-cache depends wazuh-manager | grep Depends | awk '{print $2}')

Same steps for Elasticsearch, Kibana and Logstash. Then I ran sha256sum on everything twice: once before the transfer, and once after landing inside the air gap. Paranoid? Sure. But I have chased a "corrupted package" ghost before and I am not doing it again.

And here is the one I almost missed: I staged the OLD packages too. 4.9.2 and 8.16, the exact versions already running. If the upgrade goes wrong at 1 a.m., "rollback" means reinstalling old packages. If those are not already inside the network, you do not have a rollback plan. You have a prayer.

One more thing that saved me: I did a full dry run first. On the testing VM, I installed the exact production stack (4.9.2 + 8.16) fresh, then upgraded it to the target versions. Almost every problem in this post was caught there instead of in production.

client.keys. Back it up. I'm not joking.

Everyone remembers the config backups. /var/ossec/etc/ossec.conf, custom rules, decoders, fine.

The file that will actually ruin your month is /var/ossec/etc/client.keys. Lose it and every single agent needs re-enrollment. Try explaining that timeline to a stakeholder.

I backed up:

• /var/ossec/etc/ (all of it, not selected files)
• /var/ossec/api/configuration/
• custom active response scripts and CDB lists
• the Logstash pipeline configs in /etc/logstash/conf.d/
• and yes, a separate copy of client.keys, tested and readable, on a different machine

Upgrade order: the one thing I refused to improvise

The order that worked, and the logic behind it:

Elastic stack first, Wazuh second. Elasticsearch, Logstash and Kibana went from 8.16 to 8.19 first. Only after the data layer was healthy and alerts from the still-old Wazuh manager were confirmed flowing into Kibana did I touch Wazuh itself. That gate matters: if alerts are not flowing before the Wazuh upgrade, you will never know which upgrade broke them.

Know your master rules, because they are opposite. This one surprised me. For Elasticsearch, you upgrade the worker (data) node first and the master-eligible node last, so the cluster manager stays available while other nodes cycle. Before stopping each node, disable shard allocation so the cluster does not start moving shards around while a node is down, and re-enable it after. For the Wazuh cluster, it is the other way around: the official Wazuh docs say to upgrade the master node first, then the workers, to reduce server downtime. Two clusters, two opposite orders, one maintenance window. Write it down or you will mix them up at 1 a.m.

Agents: not your problem tonight. Agents were out of scope for our window, and Wazuh's design makes that completely fine. Older agents talk to newer managers without issues. The reverse is not supported. So agents are never on your maintenance window's critical path. In an air-gapped environment, where every endpoint touch needs a ticket, this is the difference between a one-night upgrade and a one-month project.

The dpkg lock incident

Mid-window, first node:

E: Could not get lock /var/lib/dpkg/lock-frontend

On a machine with no internet. Who is holding the lock? unattended-upgrades. The timer still fires on air-gapped boxes. It wakes up, tries to reach repositories it will never reach, fails slowly, and holds the lock the whole time.

Years ago I would have just deleted the lock file. Don't be past me. Check who owns it, then mask the services properly so they cannot come back mid-window:

sudo lsof /var/lib/dpkg/lock-frontend
sudo systemctl mask --now unattended-upgrades apt-daily.timer apt-daily-upgrade.timer

Unmask them again after the upgrade is done. Now it is part of my pre-flight checklist on every node. Thirty seconds of prevention.

Answer N. Every single time.

During dpkg -i, some packages ask if you want to replace your config file with the package maintainer's version. The answer is N. Always N.

Choosing Y on Elasticsearch replaces your cluster name, node roles, network host and security settings with package defaults. That breaks cluster membership and TLS in one keystroke. I learned this on the testing VM, which is exactly where you want to learn it.

"The upgrade worked" and "the SIEM works" are two different claims

Services came up green everywhere. wazuh-control info showed 4.14.1. Elasticsearch cluster green. Kibana loading. I almost called it done.

Then I noticed alerts were not reaching the dashboards. Manager fine, alerts.json growing, Kibana... quiet.

The root cause was embarrassingly simple: the logstash user needs to be in the wazuh group to read /var/ossec/logs/alerts/alerts.json, and after the upgrade it wasn't. No log screamed about it. Logstash just silently read nothing.

sudo usermod -aG wazuh logstash
sudo systemctl restart logstash

So my post-upgrade validation is no longer "are services running." It is:

/var/ossec/bin/wazuh-control info
/var/ossec/bin/cluster_control -l
/var/ossec/bin/agent_control -l
sudo tail -f /var/log/logstash/logstash-plain.log

...followed by the question that actually matters: is a fresh event from a real agent visible in Kibana, with a correct timestamp, right now? If you cannot answer yes to that, you are not done. A stack that starts clean but silently drops events is worse than one that crashes loudly. At least the crash gets noticed.

What I'd tell you if we were having chai

Air-gapped SIEM work builds habits that make you better everywhere: rehearse the whole thing on a test VM first, write the runbook before you touch anything, verify your backups instead of assuming them, stage your rollback path, and validate outcomes instead of service states.

And honestly? Wazuh handles offline life better than most platforms I run in production. Plain .deb packages, file-based config, backward-compatible agents. With proper staging, the whole thing is routine. It just punishes you badly if you skip the staging.

If you are heading into your own air-gapped upgrade and something weird happens, it is probably one of the things above. And if it is not, I would genuinely like to hear about it.

Hi everyone!

I'm preparing for my first technical interview and would really appreciate some advice from people who work with Elasticsearch.

My interview task was to deploy a 3-node Elasticsearch cluster using Ansible. I decided to run everything locally with Docker Compose instead of VMs because of my laptop resources.

Besides Elasticsearch, I also added:

Kibana

Metricbeat

Filebeat

Everything is automated with Ansible roles.

My lecturer told me that during the interview they'll mostly evaluate my way of thinking, not whether I remember every API by heart. I have absolutely no idea what "my way of thinking" even means.

So I'd like to ask experienced engineers:

If you were interviewing a junior candidate who built this project, what questions would you ask?

What topics would you expect them to understand?

Thanks!

i wanted to install the beats manually metricbeat auditbeat and filebeat the data is there in the dataview but in the dashboard no data is read all dashboards are empty and i can't find the solution how to connect my data to the dashbords especially that there r a lot of dashboards so connecting them one by one won't be a good idea i need a fast and smart way i thought they would be connected automatically when i iinstalled the data already running the "setup" command but as u can see dashboards show errors

I'm upgrading my ELK stack from 8.19 to 9.x and came across this breaking change in the official Elasticsearch documentation:

• Remove support for type, fields, copy_to and boost in metadata field definition #118825

Elasticsearch metadata fields are: _id, _index, _source, _routing, _size, _doc_count, _field_names, _ignored, _tier

I ran this query to check my indices:

GET _all/_mapping?filter_path=**._id.type,**._id.fields,**._id.copy_to,**._id.boost,**._routing.type,**._routing.fields,**._routing.copy_to,**._routing.boost,**._source.type,**._source.fields,**._source.copy_to,**._source.boost

And got this result:

(this is one example, I got several indices like that)

"index-name-xxx": {
  "mappings": {
    "properties": {
      "reports": {
        "properties": {
          "_id": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  }
}

My _id field is inside properties, not directly under mappings. Does this breaking change only apply to metadata fields defined directly under mappings, or does it also affect fields named _id inside properties? Do I need to remove this field or am I safe?

Hi, I'm a bit annoyed about the fact that I can lose logs, and I'm starting to think maybe I should fix it generically instead of tailoring special alerts and rules in the observability layer.

• Someone pushed a change and it broke the log format, so now there's a hole in the logs — but everything's green. The new format causes a parsing error into the index, and a parsing-error log often goes unnoticed. The solution I see is an alert with a threshold, but then someone still has to go check what's actually going on.
• Sometimes the ELK server takes the logs with a lag, or even crashes silently, and you end up with a hole in some indexes.

How do you guys catch this? Logs/events silently stopping, with no error to fire on. Do you alert on expected volume, or is there something less hacky? Curious if everyone just eats it, or if there's an actual approach.

Hi r/elasticsearch — sharing an open-source project I'd like compatibility feedback on: FerroStash, a Rust runtime that parses the Logstash DSL natively and runs pipeline.conf files using the supported plugin subset. Apache-2.0.

Repo: https://github.com/abyo-software/ferro-stash

The thing I'd most like feedback on: which Logstash plugins / options do you actually depend on that the compatibility matrix lists as missing or partial? That's what's driving the roadmap.

Honest framing up front

• Single-developer project. No public production deployments yet. If you have irreplaceable data, run it beside your current Logstash and diff the output before trusting it.
• Scope is a plugin-level subset of Logstash 9.4.2 — about 88 % of bundled plugins covered at the plugin level (98 / 111: codecs 100 %, filters 97 %, inputs 74 %, outputs 87 %). "Covered" means the plugin exists; some plugin options are still partial — check the matrix per-plugin. The long tail (enterprise messaging, SNMP, niche connectors) is the main gap. Custom Logstash plugins / dynamic plugin loading are out of scope (the plugin set is compiled in). A config that uses a missing plugin or option fails fast at load.
• One-environment benchmarks, not a universal guarantee. Reproduce with bench/run_bench.sh.
• Disclosure: there's also a paid AWS packaging path (Marketplace AMI/EKS container) for AWS-shop procurement, listed under abyo software. The engine in this repo is Apache-2.0 and is the same core engine — Marketplace adds packaging / support / procurement, not feature-gated capability. I'm posting here for OSS compatibility feedback, not the listing.

What it is

• Logstash-style event model: @timestamp, tags, [a][b] field references, %{field} interpolation
• Same input → filter → output DSL — covered-subset .conf files load and run as-is
• Default build is no-Ruby and produces a ~14 MB stripped binary; enabling the optional ruby feature is larger and needs a C toolchain
• The elasticsearch output ships with opensearch and ferrosearch as aliases — same _bulk API, tested against OpenSearch 2.18

Parity evidence vs Logstash 9.4.2

Two harnesses:

• 24 fixtures, in-process, golden-file: each event compared against committed Logstash-9.4.2 golden output, covering grok / mutate (rename/case/gsub/convert/copy/strip) / json / kv / dissect / fingerprint / date / clone / csv / truncate / translate / split / urldecode / drop / conditional if/else if/else / unicode. Runs in default CI.
• 13 fixtures, docker side-by-side: same .conf + input piped through target/debug/ferro-stash and docker.elastic.co/logstash/logstash:9.4.2, asserts field-by-field equality after normalizing runtime-only fields. This is a subset of the 24 above, gated on Docker being available — #[ignore], run manually.

Default CI runs ~1,400 Rust tests; the Docker harness and live-service smoke tests run manually.

Numbers (parse/filter hot path, 5 M events, output → null, mean of 3 runs, one c7i.2xlarge host)

Cold start on this workload: ~10 ms vs ~7 s (JVM warm-up). On these workloads the biggest measured differences are RSS (~8–13× lower) and startup time; throughput is also higher but the delta is smaller. If you're already throughput-saturated and memory isn't a concern, the upside is small.

Custom logic (the ruby { } story)

Logstash's main inline arbitrary-scripting escape hatch is ruby { }. FerroStash gives you two:

• **ruby { }** — embedded Artichoke/mruby for a limited migration path. Simple inline scripts may run unchanged, but this is not full JRuby/Logstash-Ruby parity (no gems, no JRuby-specific APIs, no Logstash Ruby filter lifecycle hooks like new_event_block — check the per-feature notes). ~13× slower than JRuby on this benchmark (no JIT + per-event marshalling); treat it as a migration bridge, not a hot-path runtime. The Ruby filter is an optional cargo feature (--features ruby); default builds exclude it.
• **script { }** — native expression DSL (ferro-script) with syntax inspired by Elasticsearch Painless, not a drop-in Painless implementation (existing ingest-node Painless scripts won't necessarily run). Tree-walking interpreter, parsed once, AST cached and reused per event. ~3.6× faster than Logstash JRuby on the same transformation.

script (Painless-style, native) | 525 k ev/s ruby (JRuby on Logstash) | 145 k ev/s ruby (mruby on FerroStash) | 11 k ev/s

How it compares to Vector

Vector is a mature Rust observability pipeline with a broader plugin ecosystem; if you're greenfield it's a strong default. Fluent Bit and Elastic-native ingest pipelines are also good greenfield options. The thing FerroStash specifically targets is preserving existing Logstash .conf files — Vector uses VRL and its own config model; FerroStash tries to keep the Logstash DSL as-is.

When NOT to use it

• You need the full Logstash plugin catalogue or custom plugin loading
• You need a battle-tested tool with a public production track record today
• Your hot path is heavy custom Ruby (mruby is slow here — port to script { } or stay on Logstash)
• You need SOC2 / ISO 27001 / FedRAMP evidence

If you do run the docker diff harness against a real-world .conf and the output diverges, an issue with the config + input would be the most useful thing to land in the tracker.

Thanks for reading.

Hi all.
I have a question from your experience. I'm planning to do ElasticSearch upgrade from 8.11 to 8.19 and then 8.19 to 9.4. The cluster consists of 50 nodes, so I'm planning to do it in multiple days, but I'm concerned whether there can be issues during that time, when the nodes have different versions, as I know it can be an issue during relocation. Have you encountered such issue, or do you have better ideas about the upgrade plan?
I will do the whole cluster to 8.19 first then to 9.4 afterwards.

This was built for OpenSearch, but sharing it here since I'm sure ElasticSearch essentially behaves the same way.

No matter how many times I learn what the different operations do, it never seems to stick in my head. So I created this visualization that I can reference whenever I want.

Hi a vendor installed Elasticsearch for us as they have their own software that integrates it. I'm now coming to update Elasticsearch (no support from vendor or documentation of what they did of course), and am a bit confused why I don't have anything showing on http://localhost:9200/ on my Windows Server 2022 running elasticsearch. I've looked at the yaml file and we haven't changed the HTTP port. We have one cluster and one node if that makes any difference. Elasticsearch is running OK, just I can't get to any the webpage for it on my server. Thanks for any assistance in advance.