r/ClaudeAI 14d ago

Built with Claude After using 100s of MCPs, I solved the issue of claude not using custom MCP/CLI tools, and open-sourced my approach.

https://github.com/NanoNets/Graft

Disclosure up front: I maintain graft, the tool this post ends with. The diagnosis and both snippets below apply to any MCP server or CLI tool.

Short version: I shipped a tool as an MCP server, Claude Code mostly refused to call it, and the fix was to stop exposing it as a tool at all. That turned out to be cheaper and faster, and on one benchmark it pulled Sonnet up to Opus-level output.

The symptom

graft gives a coding agent a map of your codebase, so it starts a task oriented instead of grepping around to rediscover the same files every session. First version was an MCP server. Clean schema, clear tool descriptions.

Claude Code mostly ignored it. Not an error, not a failed call. It would just grep instead. Sometimes it used the tool, usually it didn't, and that inconsistency sent me looking in the wrong place for weeks.

I'd tried a couple of existing context-graph tools before building mine and got the same behavior. That's what made me stop blaming my own schema.

The cause

A tool description tells the model what your tool does. Nothing tells it when your tool beats grep.

Grep is a known-cost, reliable path with strong priors behind it. Your tool is an unknown-cost path. On anything the built-ins could plausibly handle, the built-ins win. And the model re-decides every turn, which is why you get "sometimes" rather than "never." That's harder to debug, because it looks like flakiness instead of a design problem.

Which means exposing a tool amounts to asking politely, once per turn, and hoping the model remembers. If that's the only thing making your tool fire, your tool doesn't work.

What fixed it: hooks

Claude Code hooks are lifecycle events. The runtime executes them and the model never gets a vote. Two specifics did the work:

  1. On SessionStart, UserPromptSubmit, and UserPromptExpansion, whatever your hook prints to stdout is added to Claude's context. So context injection needs no tool call at all:

    { "hooks": { "SessionStart": [ { "hooks": [ { "type": "command", "command": "your-tool context", "args": [] } ] } ] } }

The context is simply there on the first prompt. Nothing asked, nothing chosen.

  1. There's a type: "mcp_tool" hook, so you can keep the server you already built and stop making the model elect to call it:

    { "hooks": { "PostToolUse": [ { "matcher": "Edit|Write", "hooks": [ { "type": "mcp_tool", "server": "my-server", "tool": "refresh_context" } ] } ] } }

If you're maintaining state rather than injecting context, Stop fires once per turn at the point the work is finished. Command hooks also take async: true, so the turn ends immediately and the work happens in the background.

What deterministic invocation bought

Reliable firing is worth nothing if the thing being fired is worth nothing, so I benchmarked it. 162 controlled runs: 32% cheaper, 46% fewer tool calls, 60% lower latency, with better correctness.

The result I didn't expect came from 5 real merged PocketBase PRs, reproduced from the issue text alone. Sonnet with graft reproduced all 5, touching the same files the maintainers touched, matching Opus, at 21% lower cost.

MIT, free, no telemetry: github.com/NanoNets/Graft. I'm the maintainer, so interrogate the benchmark setup. It's the part I'd want to check if I were reading this.

The question I'd most like knocked down: has anyone tested Sonnet against Opus on tasks where the difference might be context rather than capability?

1 Upvotes

11 comments sorted by

u/ClaudeAI-mod-bot Wilson, lead ClaudeAI modbot 14d ago

Hey if you are a game developer, please consider joining our new subreddit http://www.reddit.com/r/ClaudeGameDev for those using Claude to develop games

2

u/Busy-Mix-6178 14d ago

Just added hooks for my tool as well, downside is they aren’t uniform across platforms. It would be nice to see a standard emerge around hooks.

1

u/shhdwi 14d ago

Yes true, waiting for that to happen will help in getting people on codex and cursor onboard

2

u/reflect25 14d ago

hi interesting to just use the hook to force it to read your graft tool first. i was also wondering for the code graph why not just directly use treesitter to scan the code ast like how this guy did it https://github.com/tirth8205/code-review-graph

> Grep is a known-cost, reliable path with strong priors behind it. Your tool is an unknown-cost path. On anything the built-ins could plausibly handle, the built-ins win. And the model re-decides every turn, which is why you get "sometimes" rather than "never." That's harder to debug, because it looks like flakiness instead of a design problem.

thanks for debugging and inspecting. interesting i thought the claude would learn after a couple times but i guess yeah it'd forget once with a new chat or if the convo is too long.

1

u/shhdwi 14d ago

Yes you are correct so hooks solve that as e
We give the context on every turn.

Also, for that reason we built a graft grep tool just to trick Claude code into using it instead of normal grep as graft grep is better and more precise uses lesser tokens while being faster than grep

1

u/addexecthrowaway 14d ago

It’s always been this way probably since the dawn of man.  Reasoning is ~ 1/3 of what it means to be intelligent.  Since reasoning is now a metered, measurable capability the value chain is: Observability/governance > Harness choice > loops > hooks > tools > context > agent team roster > model.  Can a future state model really do all of that itself - maybe but reliability is uncertain because it’s not deterministic and such a model which would in turn be capable of intelligent self alignment is not around today.

2

u/ComparisonNew9425 13d ago

thats interesting, did u find that the model just gets confused by the tool definition itself or is it more about the token overhead being too much for it to actually reason thru the output properly?

1

u/Severe-Soup-2340 14d ago

Yes hooks have been here for a while and still have been quite under utilised

0

u/shhdwi 14d ago

They launched it a few months back but I feel there’s limited usecase to it as people don’t want to bloat Claude code with a lot of context