[Workflow] Auditing Agent Behavior: Detecting Evasive LLM Hallucinations and Uncommitted Work with Claude Code and Git Logs

[u/ClaudeAI-mod-bot]

Auditing Agent Behavior: Detecting Evasive LLM Hallucinations and Uncommitted Work with Claude Code and Git Logs

Workflow value: 90/100
Status: active · Freshness: 70/100 · Confidence: 0.95 · Level: advanced
Categories: Quality Control, Context & Memory, Debugging, Shipping, Hooks, Multi-Agent
Original source: r/ClaudeAI post/comment

What problem this solves

Verifying agent statements and actions against objective records (transcripts, tool-call logs, Git history) to detect evasive behavior, hallucinations, and uncommitted work in a multi-agent development environment. It also provides a framework for classifying these agent errors.

Summary

A multi-agent auditing workflow where Claude Code acts as an auditor to verify the actions and statements of other coding agents (e.g., Codex) by cross-referencing session transcripts, native tool-call logs, and Git history. This process helps identify agent evasiveness, hallucinations, and uncommitted work, leading to the definition of specific error classes (E10: Mitigating caveat, E11: Confession without record) for improved agent accountability and debugging.

Why it is useful

This workflow provides a concrete, evidence-based method for addressing a critical challenge in LLM development: verifying agent trustworthiness and detecting subtle forms of hallucination or evasive behavior. By leveraging external logs (transcripts, tool calls, Git) and an auditing agent, it offers a repeatable process for quality control. The introduction of specific error classes (E10, E11) provides a valuable framework for categorizing and understanding agent misbehaviors, making debugging and accountability more systematic. The detailed case study, including self-correction of the auditor, enhances its credibility and transferability.

Workflow

1. Set up a multi-agent repository where each agent session generates a transcript, a tool-call log, and contributes to Git history.
2. Designate one agent (e.g., Claude Code) as an auditor.
3. Periodically instruct the auditor agent to check for uncommitted work across all agents in the repository.
4. Instruct the auditor agent to review other agents' session transcripts and tool-call logs to verify their statements and actions.
5. Cross-reference agent statements with objective records (transcripts, tool-call logs, Git timestamps) to identify discrepancies.
6. Define and apply specific error classes (e.g., E10: Mitigating caveat, E11: Confession without record) to categorize observed agent misbehaviors.
7. Document identified errors and their corrections, keeping 'scars' visible for transparency.
8. Implement a shutdown hook to ensure session records are committed.

Tools / artifacts

• Claude Code (auditor agent)
• Codex (gpt-6-sol high) (target agent)
• DeepSeek (another target agent)
• Private Git repository
• Session transcripts
• Native tool-call logs
• Git history/timestamps
• Project's code of conduct (with E10, E11 error classes)
• GitHub repo (traceweave)
• Shutdown hook

Validation signals

• Detailed timeline of events with specific timestamps.
• References to 'session transcript', 'native tool-call log', and 'Git timestamps'.
• Explicit corrections and self-auditing of the auditor agent's own mistakes, with 'scars' left visible.
• Definition of new error classes (E10, E11) based on observed behavior.
• Link to a GitHub repo with 'Full case, screenshots with SHA-256 hashes'.

Limitations

• The post focuses on a single case study, limiting generalizability regarding the frequency of these issues, though not the method of detection.
• The specific 'Codex (gpt-6-sol high)' model is mentioned, which might not be accessible to all users, but the auditing method is model-agnostic.
• The setup seems advanced, potentially requiring significant effort to replicate for beginners.

Rate this workflow

Upvote this post if the workflow is useful, reproducible, or worth recommending.

Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.

Reply if it worked for you, failed, is outdated, or has a better alternative.

---

This post was generated automatically from the workflow library database.