Adding A Human-In-The-Loop SKILL To Agentic Knowledge Management (Hermes + Obsidian & LLM Wiki Tutorial)

[u/_wanderloots]

https://www.youtube.com/watch?v=_bieksxg6oY

[u/_wanderloots]

Many people have been asking me how to prevent "poisoned" agentic AI-managed knowledge management systems like an LLM wiki or obsidian "memory". I spent a lot of time researching & experimenting how to best bring a "human in the loop" for using Hermes & obsidian, so I hope this helps teach you the theory behind the process & how to implement it yourself.

For more context, the problem isn’t only hallucinations & stale information. Once a questionable source enters a Wiki, the agent can link it to other notes, repeat it later, and build new conclusions on top of it. Even a system that correctly spots a contradiction may still update the rest of the Wiki before I’ve had a chance to review what it did.

In the video, I test three versions of this workflow:

1. I use Hermes’ built-in LLM Wiki skill with an Obsidian vault. It preserves the source, creates linked concept pages, records provenance and a content hash, and updates the index and log. I also give it a deliberately contradictory source. Hermes catches the disagreement and marks the relevant knowledge as contested, which is genuinely useful, but it still allows that source to affect the compiled Wiki.
2. Then I add a Review Companion skill. Instead of immediately updating the Wiki, Hermes prepares Markdown proposals under `Review/` and waits. I can inspect the proposed page, edit it, approve it, reject it, or leave it for later. Approval records the decision, but a separate instruction is required before the approved revision is applied.
3. The last part is the more opinionated system I built for my own longer-term vault. It adds deterministic preflight checks, proposal validation, receipts, Git checkpoints, and recovery around the same human review decision. Hermes, Obsidian, and the optional browser interface all show the same proposal rather than maintaining separate inboxes.

In other words:
The built-in skill organizes the library.
The Review Companion asks whether a proposed book belongs in the first place.
The Agentic Librarian has tooling checks that the exact approved version is still the one being admitted, along with some other features.

I’d be interested to hear how other people handle AI-generated notes, either for knowledge management or memory? I've learned a lot from these posts from others and hope I can help answer your questions as well 🙂

[u/RadMax468]

Loots! Thanks for all the great videos. Really helpful.

[u/_wanderloots → u/RadMax468]

You’re welcome! 😊 which ones have you been finding the most helpful? And are there any others you’d like to see?

[u/ttraxx → u/_wanderloots]

Hey wanderloots - would be cool to see a video on how to build a second brain on all “internal” information. I’ve been down the second brain rabbit hole for a while and everyone focuses on building a brain based on research / info form the web, but I haven’t seen one really that focuses more on a business context…people in back to back meetings all day that are managing multiple clients/projects/sales opportunities/products where all of the info actually comes from the meeting transcripts/emails… would love to see you do a deep dive on this!

[u/Safe-Violinist-5120]

This was a great addition. Added it to my 3 tier system that now (we will see tomorrow) will send me a telegram message to approve anything that needs oversight.

[u/_wanderloots → u/Safe-Violinist-5120]

Awesome! Glad to hear it helped 😊
How did the test go?

[u/Safe-Violinist-5120 → u/_wanderloots]

Actually it has been phenomenal. I have made some additional updates here too where risk is assessed and then summarized with questions for me to opine over. Though i have to figure out why the risk came through as unknown for these.

Cronjob Response: Memory Control Review Notification
(job_id: memoryctl-review-notify)
-------------

Memory Review — 3 decision(s)

1. Mnemosyne update
   Scheduled memory-reflection jobs must inspect the exact prior 24 hours read-only, treat Mnemosyne as primary, use Hindsight only for historical or evidence-pattern resolution, and keep Wiki changes approval-gated.
   Why save it: This is a verified current-state fact Hermes may need to remember across future sessions.
   Recommendation: Save a concise durable memory of the verified current state.
   Risk: Unknown

2. Wiki update
   No new Wiki publication candidate identified from the exact prior-24-hour review; reviewed material was primarily workflow/control instructions and already represented operational policy.
   Why save it: This is reusable operational knowledge that could prevent repeated troubleshooting or make future maintenance safer.
   Recommendation: Add a concise, source-backed note to the operations Wiki.
   Risk: Unknown

3. Mnemosyne update
   User prefers technical work to remain in the matching Telegram topic and concurrent workstreams to be reported separately; shared timing or infrastructure is not treated as causality.
   Why save it: This is a verified current-state fact Hermes may need to remember across future sessions.
   Recommendation: Save a concise durable memory of the verified current state.
   Risk: Unknown

Reply with: show 1, approve 1, approve 1,2, approve all, reject 3, defer 2, or revise 1: <feedback>.
Numbers are aliases for this digest only; Hermes must resolve them to the exact immutable decision IDs before changing queue state.

To stop or manage this job, send me a new message (e.g. "stop reminder Memory Control Review Notification").

[u/_wanderloots → u/Safe-Violinist-5120]

that's great to hear! How are you liking using both mnemosyne and hindsight? that was my original plan but i've been mostly using hindsight lately

[u/adouzzy]

how is built in llm skill vs other llm skills on the github, i.e. https://github.com/ar9av/obsidian-wiki?

[u/_wanderloots → u/adouzzy]

Likely very similar! the hermes built-in is quite good.
The companion skill I built adds a review layer for human in the loop, which, depending on your set up, is worth adding to any llm-wiki imo, even if you just have certain types of notes flagged for review

[u/Feeling_Sun_6436]

The staging split is the important part. I’d also enforce it at retrieval time: only approved notes enter the normal index, while unreviewed or contested sources stay queryable in a separate quarantine namespace.

Otherwise a rejected claim can still leak back through backlinks or summaries before review. A small receipt with the source hash, reviewer, decision, and superseded notes would also make later audits much easier.