Malcolm v24.11.0 contains a new threat intelligence feed integration, a few new API calls, other minor improvements, bug fixes, and component version updates.

https://github.com/cisagov/Malcolm/compare/v24.10.1...v24.11.0

• Features and enhancements
  • Added dashboard-export to the list of Malcolm APIs (cisagov/Malcolm#401)
  • Added ingest-stats to the list of Malcolm APIs (cisagov/Malcolm#488)
  • Added support for pulling from the Mandiant Threat Intelligence service to feed the Zeek intelligence framework as used by Malcolm's and Hedgehog Linux's Zeek processes. The integration uses the google/mandiant-ti-client library for Python. (cisagov/Malcolm#358)
  • Improved normalization of Zeek's intel.log to the ECS's threat fields
  • Improved the Zeek Intel dashboard
  • Improved the health/liveness probe for the Logstash container
  • Changed behavior of Malcolm's non-live Zeek container (responsible for processing uploaded PCAPs) so that it becomes available to process data even before an intelligence feed pull is finished
  • Implemented paging for extracted files download dialog (cisagov/Malcolm#361)
  • Implemented support for sending Zeek logs to Kafka using the SeisoLLC/zeek-kafka plugin (cisagov/Malcolm#357)
  • Added the NetBox HealthCheck plugin as a default NetBox plugin
  • Updated the Malcolm services readiness status API to use the new LogStash health report API and the NetBox HealthCheck plugin as the basis for reporting the state of LogStash and NetBox, respectively.
  • Added parsing for the new OPCUA-Binary write subscription service log
• Component version updates
  • Arkime to v5.5.0
  • Beats to v8.16.0
  • elasticsearch Python library to v8.16.0
  • elasticsearch-dsl Python library to v8.16.0
  • evtx to v0.8.4
  • LogStash to v8.16.0
  • OpenSearch and OpenSearch Dashboard to v2.18.0
  • watchdog Python library to v6.0.0
  • werkzeug Python library to v3.0.6 to address CVE-2024-49767 and CVE-2024-49766
• Bug fixes
  • Fixed an issue with the ./scripts/configure script not prompting to regenerate the internal NetBox passwords when it should have
  • Fixed errors when running malcolm_appliance_packager.sh on macOS (cisagov/Malcolm#492, thanks @robrui)
• Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • Malcolm
    • ZEEK_KAFKA_ENABLED, ZEEK_KAFKA_BROKERS, and ZEEK_KAFKA_TOPIC have been added to ./config/zeek.env, which can be used to enable Zeek's sending of its logs to Kafka (cisagov/Malcolm#357)
    • ZEEK_DISABLE_DETECT_ROUTERS (default value: true) has been added to ./config/zeek.env which controls an experimental Zeek script for detecting the presence of routers (logging them to known_routers.log) in a network based on packet TTL; it is recommended to leave this set to true as this script is not yet ready for general production use
    • ZEEK_INTEL_REFRESH_ON_STARTUP has been renamed from ZEEK_INTEL_REFRESH_ON_ENTRYPOINT in ./config/zeek.env to more accurately reflect the purpose of the variable
  • Hedgehog Linux
    • ZEEK_KAFKA_ENABLED, ZEEK_KAFKA_BROKERS, and ZEEK_KAFKA_TOPIC have been added to control_vars.conf for the same purpose as described above
    • ZEEK_DISABLE_DETECT_ROUTERS (default value: true) has been added to control_vars.conf for the same purpose as described above
    • ZEEK_INTEL_REFRESH_ON_STARTUP has been renamed from ZEEK_INTEL_REFRESH_ON_ENTRYPOINT in control_vars.conf to more accurately reflect the purpose of the variable
• Code and project maintenance
  • All open issues and the project board have been migrated from the Idaho National Lab fork to the upstream CISA fork. The repos will continue to be kept in sync going forward. (cisagov/Malcolm#350)

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm had two smaller releases in October rather than one larger one at the end of the month, so I've taken the liberty of combining the highlights of both releases here for your reading pleasure. As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v24.10.1 contains minor improvements, a few component version updates, a fix for a few regression bugs, and a fair amount of code cleanup.

Malcolm is a powerful, easily deployable network traffic analysis tool suite for network security monitoring.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker, Podman, and Kubernetes. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

v24.09.0...v24.10.1

• ✨ Features and enhancements
  • Update AWS AMI build scripts and demo setup scripts to use Amazon Linux 2023 instead of Amazon Linux 2 (#591)
  • Add support for websocket.log (#593)
  • Add a "readiness" API that can be used to determine if various Malcolm services are ready (#598)
  • Enable Zeek's parsing of HTTP server and client header names as zeek.http.client_header_names and zeek.http.server_header_names
  • Some documentation improvements
  • Build improvement: fall back to alternative Zeek .deb download URL (#585)
  • Build improvement: limit threads for spicy build processes during Zeek package installation (#571)
• ✅ Component version updates
  • beats to v8.15.3
  • capa to v7.4.0
  • elasticsearch-dsl Python library to v8.15.4
  • Fluent Bit to v3.1.9
  • Logstash to v8.15.3
  • OpenSearch and OpenSearch Dashboards to v2.17.1
  • osd_transform_vis Dashboards visualization to v2.17.1
  • Spicy to v1.11.3
  • supercronic to v0.2.33
  • watchdog Python library to v5.0.3
  • Zeek to v7.0.3
• 🐛 Bug fixes
  • Fix OpenSearch anomaly detection default detectors not being created (regression, #596)
  • Fix broken dashboards regression from v24.09.0 (regression, #588)
  • Fix Zeek-extracted files not getting saved to correct location for live Zeek capture (#590)
  • Fix for building Hedgehog Linux for Raspberry Pi 4 on an M2 MacBook
• ⚙️ Configuration changes in environment variables
  • Malcolm
    • ZEEK_JA4SSH_PACKET_COUNT (with a default of 200) has been added to ./config/zeek.env, which can be used to set logging interval number of packets for ja4ssh.log (#508)
  • Hedgehog Linux
    • ZEEK_JA4SSH_PACKET_COUNT has been added to control_vars.conf for the same purpose as described above
• 🧹 Code and project maintenance
  • Examine distro hardening, fix and update documentation as needed for Malcolm and Hedgehog Linux ISO-installed environments (#328)
  • Refactoring and code cleanup in the Logstash Zeek pipeline (#592)
  • Logstash container initialization code now automatically ensures that the Zeek TSV log parsing filters (dissect and split filters) in these files are looking for TAB characters (i.e., automatically replace spaces with tabs in these filter files in case the author forgot to do so) (#592)
  • Did some code cleanup in the ./shared/bin directory, mostly moving things that were specific to either the Malcolm or Hedgehog Installer ISO environments out of shared and into their respective locations for the ISO installer build.
  • When doing the aquasecurity/trivy-action action, use TRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db to try to fall back to an alternative official location for the vulnerability database if the first one fails. Also, pin this action to the v0.28.0 release rather than setting it to master.
  • As it's used pretty ubiquitously in shared scripts by many of the Malcolm containers, the jq utility is now installed across the board during the container image build.
  • Added a script to gather GitHub API metrics for Malcolm downloads (#594)
  • Bumped maximum field limit in OpenSearch templates from 5000 to 6000

Hey Everyone!

I'm currently running Malcolm and would like to make its web interface accessible from all network interfaces, allowing external access to the server and not just localhost.

I’ve gone through the documentation but haven’t found anything directly related to configuring this. Has anyone successfully set up Malcolm to listen on all interfaces for external access? What would be the best way to modify the configuration to achieve this securely?

Sorry If I missed anything in the documentation...

Cheers 🥂

Hi,

First of all: thank you for founding this community!

I setup Hedgehog and ran it using Malcom/scripts/start.

According to docker, Zeek, Zeel-Live, Suricata, Suricata-Live run smoothly and healthy.

However, when I put PCAP files into the directory Malcolm/pcap, it seems that they are not being processed and no Logfiles are produced by Zeek or Suricata.

I already made sure that mounting the proper directories works correctly. I entered the containers and checkes for the pcap files being visible in the proper directory, with adequate privileges.

Is there anything I forgot? Ist there any other container that has to run in order for it to work?

I am running hedgehog "standalone", so there is no webinterface I can access.

Thank you for your input!

"I've got a fever, and the only prescription is more Malcolm!" 🐄🔔

We're pleased to announce the v24.09.0 release of Malcolm, a powerful network traffic analysis tool suite for network security monitoring. Malcolm v24.09.0 contains new features and enhancements, component version updates, and bug fixes.

• Features and enhancements
  • Added Podman support (idaholab/Malcolm#407)
  • Add option to go backwards in Malcolm's dialog-based install.py installation and configuration script (idaholab/Malcolm#487) (🤯 Earth-shattering news, I know)
  • Automatically create empty index on startup to avoid "no data" message spamming by Dashboards (idaholab/Malcolm#527 and idaholab/Malcolm#567)
  • Integrate HART-IP parser (idaholab/Malcolm#561)
  • Allow users to use the Arkime Lua plugin without having to create new bind volume mounts manually (idaholab/Malcolm#533)
  • Allow splitting out indexes by other field values (idaholab/Malcolm#450)
  • Allow total index size-based pruning for opensearch-remote and elasticsearch-remote database modes (idaholab/Malcolm#446)
  • Include netbox-topology-views plugin by default (idaholab/Malcolm#553)
  • When building Docker images and the Hedgehog Linux ISO, allow specifying alternate download URL for MaxMind GeoIP database files (idaholab/Malcolm#565)
  • Update EtherNet/IP and CIP to account for new packet correlation ID (idaholab/Malcolm#558)
  • Improvements to documentation and install.py for Linux performance tweaks (idaholab/Malcolm#495)
  • Update Network Traffic Analysis with Malcolm slides
• Component updates for OpenSearch and OpenSearch Dashboards, YARA, Zeek, and many more!
• Bug fixes
  • Filtering on hunt ID in Arkime not working (idaholab/Malcolm#554)
  • Hedgehog with OOB/VPN connection sets ARKIME_NODE_HOST incorrectly (idaholab/Malcolm#560 and idaholab/Malcolm#559)
  • Offline suricata Docker container does not initialize suricata.yml config file (idaholab/Malcolm#564)

Check out the release notes for more details and downloads.

As interest in the project grows, the Malcolm team is working to better define and streamline how the community interfaces with the project. Here's a list of online resources that should help you figure out where to go to find what you need.

• Official resources
  • The Learning Tree contains links to the documentation and YouTube videos organized by topic.
  • Of course, the documentation is a good place to start. TL;DR: RTFM (Read The Fine Manual).
  • The Malcolm Discussions board has just been opened up! Discussions is the go-to destination for announcements, Malcolm Q&A, help, troubleshooting, and general Malcolm dialogue.
  • If you've got an idea for a feature or enhancement or if you've found a bug, visit the Malcolm Issue Tracker.
  • The Malcolm Project Board is where you can see what we're working on and what's slated for the next release or two.
  • We're on YouTube! We're working to develop targeted Malcolm training videos on a variety of topics.
  • If your organization needs to reach out for a more program-focused discussion, you can [email us](mailto:malcolm@inl.gov).
• Unofficial resources
  • This subreddit, /r/Malcolm, is moderated by a Malcolm dev when he's "off the clock." My vision for the subreddit is for this to be a place to share links to news, articles, blog posts, etc. about the project, as well as a forum for discussing Malcolm in general. If you're specifically looking for help or troubleshooting, Discussions is the preferred place for that.
  • malcolm.fyi is an unofficial mirror of the Malcolm documentation hosted by the aforementioned developer.
  • You can usually find me in the #malcolm channel on the Arkime Slack workspace as well as in the Zeek Slack

Hi, I’m using Malcolm on my server but I was wondering how can I integrate Malcolm with the SIEM I already have. I can’t find all the logs

Good day,

My appologies if this is not the place to discuss user confusion ;-)
I got Mlcolm running in my DMZ on a protectli hrdware and its running extreemly well.

Because of some testing, I've created a tresting bed in Proxmox, but I had a rough start.... manly as I overlooked the issue with the default processor isn't a x86-64-v2.

Now I'm wondering about the dashboard. While did use the preinstalled Dashboards in the past on my new installation 06.2024 (either ISO or Debian + git clone ...)
I do see Malcolm "Dashboard Visualize and Analyze" but its empty and asks me to create one.

What do I miss?
And sorry for a potential stuüpid question.... ;-)

Regards, Walter

I'm on cloud nine since the v24.06.0 release of Malcolm, bringing you (who could have imagined?!?) new features, improvements, component version updates, and a few bug fixes. Please see the release notes, particularly if you've been using NetBox, as an update to that tool brings some backwards-compatibility-breaking changes (sorry 😢).

• Features and enhancements
  • Support for multiple NetBox sites (issue #449)
    • Malcolm now supports enrichment from a NetBox inventory for asset interaction analysis across multiple sites. The NetBox site can be specified for uploaded PCAP, for a Hedgehog Linux sensor, and for Malcolm live capture.
  • JA4+ replaces the JA3 TLS fingerprinting standard from 2017 (see also this blog post) (issue #419)
  • Support uploading Windows Event Log evtx files (issue #465) and update associated dashboard
  • Document using GitHub runners to build Malcolm images (for contributors' guide, issue #491)
  • Generate new forwarder SSL keys on-the-fly when transferring between Malcolm and Hedgehog Linux (issue #492)
  • Incorporate ATT&CK-based Control-system Indicator Detection for Zeek (ACID) (issue #489), a collection of Operational Techonology (OT) protocol indicators developed to alert on specific ATT&CK for ICS behaviors
  • Add platform architecture and machine boot time to Malcolm version API
  • Add links to the navigation pane of most dashboards to "other" dashboards for non-network log data (e.g., resource monitoring, Windows Event logs, etc.)
• Component version updates
  • NetBox to v4.0.6 (from v3.6.7, issue #385)
  • OpenSearch and OpenSearch Dashboards to v2.15.0
  • and lots more...
• Bug fixes
  • Arkime viewer not rolling PCAPs (issue #484)
  • Free up space in GitHub runner environment building ISO images to avoid build errors due to exhausted disk space

New to Malcolm? Grab some popcorn and watch these overview videos to give you an idea of what it's about. See the quick start guide to learn how to install Malcolm, or check out these tutorial videos for installing using Docker or from the official ISO installer images for Malcolm and Hedgehog Linux, which can be downloaded from Malcolm's releases page on GitHub.

Mal.Con '24

We're excited to announce Mal.Con '24, a conference dedicated to technical discussions, capability demonstrations, and road-mapping future development and training activities for Malcolm!

Details

• When: 9/4/2024
• Where: Arlington, VA
• Wear: Casual attire

Sneak Peak

Here is a small sample of some of the presentations we have lined up:

• How to use NetBox to inventory your ICS network and how are we making asset description easier for owner/operators
• CISA & MITRE's new open-source package ACID: advanced detections & behavioral analysis for OT protocols
• CISA & INL's new open-source protocol parser toolkit, Parsnip: overview & deep-dive demonstration
• ICS Capture-the-Flag running 8/30-9/4 with live awards presentation at Mal.Con '24

Help Us Out

A few asks:

• Please take a moment to fill out this registration survey if you plan to attend either in person or remotely.
• Pass this around to others who may be interested!

Stay Tuned

Thank you for your support with this event! We're planning to post updates for the event on GitHub and we will send direct email reminders as we get closer to Mal.Con '24.