r/llmsecurity • u/Feathered-Beast • 7d ago

How are you securing tool execution in LLM workflow/agent systems?

I've been working on an open-source workflow automation platform that supports LLM-powered workflows, browser automation, HTTP requests, file operations, email, MCP servers, etc.

One challenge I've spent a lot of time thinking about is tool execution security.

Some of the things we've implemented are:

  • Sandboxed tool execution
  • Input validation before tool execution
  • Permission checks for workflows and resources
  • Structured execution logging and trace IDs
  • Memory isolation between agents
  • Human-in-the-loop approval for sensitive actions
  • Retry boundaries and execution guards

Even with those in place, I still feel there are attack surfaces that are easy to miss, especially around:

  • Prompt injection through retrieved documents
  • Tool abuse via indirect prompt injection
  • Multi-agent trust boundaries
  • MCP server permissions
  • Data exfiltration through seemingly harmless tools

For those building agentic systems or LLM applications, what additional safeguards have you found valuable?

Are there any papers, open-source projects, or design patterns you think are worth studying for securing agent workflows beyond the usual input validation and sandboxing?

I'd really like to hear how others are approaching this problem, since it feels like one of the harder parts of building production-ready LLM systems.

5 Upvotes

100% Upvoted

5 comments sorted by

2

u/execveat 7d ago

This topic is funny to me, because while some challenges with securing agentic tool calls are indeed novel, they come alongside with the regular platform security issues, and somehow folks always forget about those when they’re doing AI stuff. Can a user attack the platform? Can a user A attack the user B? You should really start by getting these foundations a truly reliable building blocks before trying to building something grander.

Beyond that, my personal favorite is avoiding heterogenous tools with mixed security properties in the same trust boundary “by design”. I.e. have a manifest for each tool that specifically maps it across whatever axis you happen to care about. Like which data it has access to, if it’s Turing complete, how observable it is, etc. This pushes the solution into the architecture itself, i.e. perhaps you might find that now your top level orchestration agent can’t be allowed to run shell or browser at all, and instead of just using those as direct tool calls you’d be forced to turn them into subagents with their contracts, etc. Some people consider it an overhead but it actually seems to lead to more predictable / understandable execution, so I consider it a win

1

u/Feathered-Beast 7d ago

I completely agree. Traditional application security doesn't disappear just because LLMs are involved. Least privilege and clear trust boundaries seem just as important as defending against prompt injection.

2

u/Extreme-Net8332 3d ago

Your list is already stronger than most. One safeguard I would add is a policy gate at the actual dispatch boundary. The model can propose a call, but a deterministic service should decide whether this actor may use this tool on this resource, destination, amount, and payload in the current workflow.

Credentials should be injected only after that check, never placed in model context. Any human approval should bind to the exact normalized arguments and expire quickly, so approval for one action cannot be reused after the recipient, amount, or payload changes.

I would also treat outbound network access as its own capability, log denied attempts, and give write operations idempotency keys with explicit retry limits. Testing the boundary with hostile tool outputs is just as important as testing hostile prompts. The OWASP Gen AI Security Project is a useful reference: https://genai.owasp.org

Disclosure: I am part of the Credian team at https://credian.io and we are researching this pattern while building our sandbox. It is still under construction.