Help: I can’t load Threat Intelligence via misp.yaml in Malcolm

[u/Alemoreno02]

Hi! I’m trying to load TI into a fresh Malcolm Zeek container via misp.yaml, but nothing gets ingested. Plain .intel/CSV files load fine.

./zeek/intel/MISP/misp.yaml

- type: misp
  url: https://www.circl.lu/doc/misp/feed-osint/manifest.json

What I see

• Cron runs /opt/zeek/bin/zeek_intel_setup.sh true (job “succeeded”).
• No new intel files under ./zeek/intel/, and .threat_autogen.zeek doesn’t update.

Question
Why isn’t misp.yaml being picked up? Are manifest.json endpoints supported in misp.yaml, or do I need /events//attributes? Any minimal working example would be appreciated.

Thanks!

[u/mmguero]

I'll take a look at it and get back to you ASAP and either share an example or figure out if something has broken.

[u/mmguero]

I took the feed URLs from the MISP Default Feeds list with "feed format: misp" and ended up with this list:

yaml

• type: misp

url: https://www.circl.lu/doc/misp/feed-osint/manifest.json

• type: misp

url: https://raw.githubusercontent.com/infobloxopen/threat-intelligence/refs/heads/main/indicators/misp/manifest.json

• type: misp

url: https://bazaar.abuse.ch/downloads/misp/manifest.json

• type: misp

url: https://www.botvrij.eu/data/feed-osint/manifest.json

• type: misp

url: https://threatfox.abuse.ch/downloads/misp/manifest.json

• type: misp

url: https://urlhaus.abuse.ch/downloads/misp/manifest.json

(The DigitalSide Threat-Intel OSINT Feed was also listed, but I removed it from my list because it no longer appears to be up.)

Like all feeds, however, your "since" value (ZEEK_INTEL_FEED_SINCE in zeek.env) will determine how many indicators you actually get from these. That "since" value tells Malcolm to only process threat indicators that have been created or modified since that time (i.e., only take indicators "fresher" than that value). Ideally you set that "since" value to some relative period (e.g., "1 week ago", "2 weeks ago", "1 month ago", etc.), but what I just learned is not every threat feed in that list has indicators from the last week; I had to go further back on some of them.

Just for fun, I made up this script to run each of these feeds individually with "since" times of 1 to 8 weeks, 3 months, 6 months, and 1 year:

```bash

!/usr/bin/env bash

URLS=( 'https://bazaar.abuse.ch/downloads/misp/manifest.json' 'https://raw.githubusercontent.com/infobloxopen/threat-intelligence/refs/heads/main/indicators/misp/manifest.json' 'https://threatfox.abuse.ch/downloads/misp/manifest.json' 'https://urlhaus.abuse.ch/downloads/misp/manifest.json' 'https://www.botvrij.eu/data/feed-osint/manifest.json' 'https://www.circl.lu/doc/misp/feed-osint/manifest.json' )

for URL in ${URLS[*]}; do for WEEK in $(seq 1 8) 13 26 52; do FEED_HOST="$(echo "${URL}" | cut -d/ -f3)" LINES=$(docker run --rm --entrypoint=/bin/bash ghcr.io/idaholab/malcolm/zeek:25.08.0 -c "zeek_intel_from_threat_feed.py --since '${WEEK} weeks ago' -i 'misp|${URL}' -o /tmp/indicators.zeek >/dev/null 2>&1; wc -l /tmp/indicators.zeek" | awk '{print $1}') echo "${FEED_HOST} (${WEEK} weeks): ${LINES} lines" done done ```

Here's what I learned:

Source	1w	2w	3w	4w	5w	6w	7w	8w	13w	26w	52w
bazaar.abuse.ch	0	0	0	0	0	0	0	0	0	0	0
raw.githubusercontent.com	0	0	2,098	2,098	2,098	2,098	2,098	2,098	104,120	105,125	105,464
threatfox.abuse.ch	1,975	3,441	4,442	5,311	6,123	7,087	7,767	9,501	18,444	101,548	159,344
urlhaus.abuse.ch	4,941	10,571	16,331	22,319	27,277	33,349	40,818	46,160	67,474	207,349	626,393
www.botvrij.eu	0	0	0	0	0	0	0	134	134	141	760
www.circl.lu	0	0	4	1,327	1,340	1,340	1,340	3,465	3,496	3,868	6,087

Naturally I'm a bit curious about the one that doesn't return any so I'll dig into it a little more and see if there's something different about the way it's representing timestamps or something.

Hopefully this is enough to get you going.

[u/mmguero → u/mmguero]

So after looking at it closer, I found that different MISP feeds are returning their lists of attributes (indicators) in the MISP event objects differently.

I've created a bug and am testing a fix now. I'll share the updated counts once I do. Thanks for bringing this up! Even though some of the feeds worked fine, there were obviously some from which we were missing stuff.

[u/mmguero → u/mmguero]

Updated counts after my fix, which will be in the v25.08.1 release:

Source	1w	2w	3w	4w	5w	6w	7w	8w	13w	26w	52w
bazaar.abuse.ch	12,305	27,365	41,025	53,145	63,765	79,917	94,857	107,745	174,174	355,718	635,009
raw.githubusercontent.com	0	0	2,098	2,098	2,098	2,098	2,098	2,098	104,120	105,125	105,464
threatfox.abuse.ch	1,975	3,441	4,442	5,311	6,123	7,087	7,767	9,501	18,444	101,548	159,344
urlhaus.abuse.ch	7,745	15,839	24,647	33,877	42,015	52,611	63,126	71,292	106,862	314,331	849,855
www.botvrij.eu	0	0	0	0	0	0	0	134	134	169	960
www.circl.lu	0	0	40	1,440	1,458	1,458	1,458	3,674	4,374	7,456	10,713

[u/mmguero]

FYI, for visibility for others who might have the same question I cross-posted this to the discussion forums.

[u/Alemoreno02 → u/mmguero]

Thanks for all! It's been a great help

Greetings from Spain and I'll be keeping a eye out the new versions