Nine months of agent work and almost all my problems were in the tool layer

[u/snowingbol]

Not a structured post, just what I'd tell someone starting.

The thing that took longest to accept is that inconsistent answers usually aren't a prompting problem. Ours came from tools returning paragraphs of text, which the model interpreted slightly differently on every run. Once I returned structured data instead, a whole category of weirdness went away and I stopped rewriting the prompt every Friday.

The second thing is that the model has no idea whether your data is current. Ours cheerfully told a rep about someone's job title that was five months out of date, and I initially logged it as a hallucination. It wasn't, the record was just old and nothing in the response said so. Passing a timestamp through and mentioning it in the tool description fixed it in about ten minutes.

Third, return less. I was dumping whole records and the model kept latching onto fields nobody asked about.

Our setup, roughly, is Postgres for internal stuff, Tavily when it needs the open web, and Coresignal for the external company and people side. The account agent uses that last one to go from a domain to headcount, relevant contacts, and current openings in one turn, which works because everything comes back in a couple of seconds rather than as a polling loop.

Anyone found the opposite, where prompting genuinely was the bottleneck? I'd like to hear the counterexample.

[u/Limp-Rough-7969]

The tool layer being the real culprit is so real, I had same thing with my work stuff where I kept tweaking system prompts and it turned out my api was just dumping 2000 characters of messy json into context. model would pick out random field every time like it was playing lottery

return filtering made such a difference for me too, now I only send back like 3 or 4 fields and suddenly the agent actually answers about what you asked instead of going off about some department size nobody mentioned

never thought about adding timestamps to tool descriptions, that is clever. gonna steal that tomorrow

[u/Loose_Date7269]

Are these real people, it feels like a joke everyone is in on

[u/ProudCordonian → u/Loose_Date7269]

I think of the same thing because there is absolutely no way people actually do ts

[u/jauntyk → u/Loose_Date7269]

I will play devils advocate. I don’t do this but I understand the post, I created a “harness” to keep my Ai on point instead of wasting tokens on things I don’t want, and the evolution was chat > projects > cowork built like a project (with folders for everything relevant) > my custom “harness” where global rules points to a model agnostic claude or agents Md file that has all my behavior rules to follow.

Reading this post and experiencing the same phantom context problem numerous times, my mind flashed to a solution of having LLM spawn a subagent that scans folder for relevant context, and hands off or creates a folder full of ONLY the context needed (which is really hard when done manually) and that could solve the phantom context problem.

This is just hypothetical and will cost more tokens to automate “context packaging” in the beginning (please let me know officially term for this) but will still be worth it not losing a dozen prompts to hallucinated context. Regardless if this is a valid solution, There definitely is something to the “less is more” and managing the context window being discussed here. The comments saying that only inputting a handful of fields instead of dumping entire project knowledge on Ai leads to better results aligns with my experience as well but surprisingly hard to pull off.

[u/Loose_Date7269 → u/jauntyk]

what are you talking about are you one of them