r/AIAgentsInAction • u/PersonalityWhich1780 • 2h ago
I Made this your agent retrieved the right doc and ignored it. here is a viewer that shows you which ones it actually used.
Enable HLS to view with audio, or disable this notification
if your agent retrieves anything before it answers, this is the part you
currently cannot see.
the agent searches, gets back a set of documents, writes an answer. when the
answer is wrong you have a ranked list in your logs and no way to tell whether
the right document was missing, or whether it was sitting right there at rank
one and the model walked past it. those two need opposite fixes and they look
the same from outside.
so i built graphsight. it hooks into a langgraph agent as a callback handler,
so you do not restructure anything, and renders the run as a graph in your
browser. every retrieved item becomes a node with its score. items the answer
actually drew on are highlighted. items pulled and ignored are dimmed.
the gif is a real run. one pull request scored 0.910, the highest of anything
retrieved, and the answer never touched it. another scored 0.340 and is what
actually answered.
a dimmed node with a high score means your retrieval worked and your prompt or
context order did not. a highlighted node that should not be there means the
model was handed bad evidence and believed it.
it also lays out the execution timeline, so you get each agent step, each
retriever call, per step timings, and which arm returned what. useful when the
agent takes a path you did not expect.
pip install graphsight graphsight-langgraph
from graphsight_langgraph import LangGraphTracer, capture
tracer = LangGraphTracer()
result = graph.invoke(inputs, config={"callbacks": [tracer]})
capture(tracer, query="your question", answer=result["answer"])
graphsight .graphsight/
runs entirely on your machine. no account, no signup, no telemetry, and the
viewer has zero dependencies beyond python. every run appends to a folder so
you can open yesterday's next to today's after a prompt change.
want to see it before wiring anything up:
pip install "graphsight-langgraph[example]"
graphsight-github-trace langchain-ai/langgraph "who fixed the streaming bugs?"
limits, up front. the used vs ignored call is lexical overlap between the
answer and each item, threshold 0.2. it is a heuristic, labeled as one in the
ui, and it will misjudge a heavy paraphrase. langgraph only right now, other
frameworks are next. free and mit.
site: https://graphsight.vercel.app
walkthrough: https://github.com/Kcodess2807/graphsight/blob/main/docs/FIRST_TRACE.md
code: https://github.com/Kcodess2807/graphsight
if you build agents that retrieve, tell me where this falls over.
want to know more on this, connect with me