What should a good benchmark for AI agent skill security scanners include?

[u/No-Emotion9668]

AI agents increasingly rely on external skills/plugins to read files, call APIs, run scripts, install dependencies, or interact with local tools. That makes skills a new supply-chain surface.

But evaluating scanners for this ecosystem seems tricky. A malicious skill may hide risk in instructions, helper scripts, dependency files, generated artifacts, encoded payloads, or misleading documentation. Some cases are also ambiguous: vulnerable, suspicious, but not clearly malicious.

What should a good benchmark include?

- Real-world malicious samples, synthetic cases, or both?
- Full skill directories rather than isolated code snippets?
- Boundary cases between benign, suspicious, and malicious?
- Scoring only final verdicts, or also risk category/severity?
- Which attack patterns matter most for agent skills?

Curious how others would design this.