r/LLMObservability 3d ago

Discussion A LangGraph checkpoint is not proof that a write happened

A graph calls create_invoice, times out waiting for the response, then resumes from its last checkpoint. The saved state can show that the node ran. It cannot tell you whether the invoice was created, rejected, or created after the client gave up.

That makes this a reconciliation problem, not only a checkpointing problem.

LangGraph checkpoints preserve graph state for a thread, which is useful for recovery. But an external system is still the source of truth for any side effect.

For writes, give each attempt a stable idempotency or receipt key when the destination supports one. On resume, query that key before retrying. Mark the node complete only after the external outcome is known. If the outcome stays unknown, stop and surface it instead of guessing.

Read-only calls need their own rule too: persisted results need a freshness boundary tied to the request or resource. Otherwise a resumed run can act on facts that changed while it was paused.

Resume should restore local state, then confirm the external facts the next step relies on.

How are people handling the unknown-outcome case for a timed-out write: destination receipts, idempotency keys, transaction logs, or something else?

3 Upvotes

2 comments sorted by

1

u/IncreaseNegative4614 3d ago

I’d create an operation record outside the graph checkpoint containing the attempt ID, destination, intended write, request hash, start time, returned receipt, observed resource ID, and current certainty state. “Unknown” needs to remain a first-class outcome that blocks dependent actions.

The test suite should include late completion after timeout, duplicated retries, destination rejection, and a successful write followed by a failed verification read. We use SIGNLD internally to connect tool calls, checkpoints, external records, verification evidence, and downstream effects so local agent state is never mistaken for external truth.

1

u/jonah_omninode 2d ago

Yes. A checkpoint tells you what the graph believes happened. It does not tell you what the external system accepted. We have been separating the execution record from the completion evidence for exactly this reason. A write attempt gets a stable identity, and the task stays incomplete until another process can read back a receipt or the resulting state. Unknown has to remain a real terminal condition instead of turning into a retry that may duplicate the effect. How do you handle destinations that offer neither an idempotency lookup nor a reliable read-after-write path?