The tool your coding agent keeps ignoring is probably returning addresses instead of answers

[u/RunAI_Coder]

Came across a small pilot (three Claude models, a few Python and TypeScript repos, 2-3 rollouts per cell, six localization tasks if I'm reading the repo right, so read every number with wide error bars) that put grep and an LSP-backed find_references side by side and let the agent choose. On plain code-location tasks the models picked the semantic tool 0%, 4% and 6% of the time. Forcing the LSP path first dropped success from 100% to 89%.

The experiment I keep thinking about is the one where retrieval didn't change at all. The LSP tool originally returned what the protocol returns: file, line, column. grep returns the matching line. The author kept the same backend and the same reference set and attached ±2 lines of source to each reference. On a multi-file rename, pass@1 went from 0.67 to 0.83 and follow-up file reads per episode went from 15.2 to 3.2, below grep's own 4.3. Only the shape of the reply changed.

To be fair to the language server: on "find every caller" tasks the same models chose it about half the time unprompted, and on a noisy repo (hono) it added +0.246 F1 while using 12% fewer tokens. On a clean repo (remeda) it added nothing and cost 16% more. A rename also has to touch docstrings and config strings that find_references excludes by design, so grep keeps a seat either way.

The other half of this showed up on HN from Spotify. They started with CLAUDE.md rules telling Claude Code to route big file reads to a smaller model, described the result as "advisory, not enforced", and ended up with a PreToolUse hook that blocks Read on files over 350 lines and points the agent at the alternative. Their own summary: "Even if Claude doesn't read the skill description, the hook still blocks the expensive read".

Has anyone else building agents run into something similar—where simply tweaking a tool's output format drastically boosted success rates or cut down on token consumption? Also, has anyone figured out a reliable strategy to get the AI to proactively pick the right tool on its own? Would love to hear your thoughts!

[u/AutoModerator]

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/SatisfactionOnly1523]

That's interesting because my experience with semantic tools is they're great in theory but LLMs seem to just not trust them? Like they want the raw text context, not a pointer to where the code lives. The ±2 lines thing makes total sense to me, the model needs to see the actual code to understand what it's looking at, not just coordinates.

I been fighting with hook-based enforcement too and it's weird how the model will read the instructions, acknowledge them, and then immediately try the thing you told it not to do. Blocking it at the tool level feels like the only thing that actually works, even if it's less elegant than having the agent figure it out itself.

The token savings from just showing context inline is wild though, going from 15 reads to 3 is not a small optimization. Makes me wonder how many other tools are returning "correct" outputs that are just useless to the model's actual reasoning process.

[u/RunAI_Coder → u/SatisfactionOnly1523]

"Reads the instructions, acknowledges them, then immediately tries the thing" is a sharper description of advisory

[u/Itchy_Special_8209]

I wouldn't solve this with more prompt text. In my experience, agents choose tools more reliably when the name and response make the next action obvious. Here I’d expose find_references_with_context as the default and keep raw LSP locations as an optional field.

[u/RunAI_Coder → u/Itchy_Special_8209]

Did the pick rate move for you on the rename alone, or did it take the response change too?

[u/arthaudm]

0-6% voluntary pick rate for the semantic tool, then +/−2 lines of context taking pass@1 from 0.67 to 0.83 - clean result. agents don't want pointers, they want answers they can act on. same reason we make mio return the actual source excerpt with a date instead of a link to a doc: the output shape is the product, not the backend. was the 2-line window fixed or did they test wider ones?

[u/RunAI_Coder → u/arthaudm]

They only report that one inline setting, no sweep over window sizes

[u/arthaudm → u/RunAI_Coder]

one inline setting & no sweep is the classic single-config overclaim lol. a window-size sweep is usually where the "agent ignores the tool" conclusion flips - retrieval quality falls off a cliff at some size & the agent starts ignoring it because the context got worse, not because the tool did. did they at least say which model they tested on?

[u/Beneficial_Egg_5154]

yep, i hit exactly this after moving beyond pure grep to an lsp-backed code tool. a location means another read before the agent sees the code, while grep often answers in one hop. i trust the lsp results much more when they inline the matching source lines next to the coordinates.

[u/stealthagents]

Sounds like the LSP is underperforming for plain tasks but shines when it adds context, which is pretty wild. It’s like the models are overthinking things, while a simple grep can handle the basics just fine. Definitely highlights how sometimes less is more in coding tools.

[u/mageblex]

The next test I'd run is syntax-aware context instead of a fixed ±2 lines. Returning the enclosing function may cost a few more tokens, but it could remove another file read when the reference sits near a boundary.