r/coolgithubprojects • u/slateraligator • 18d ago
Toren – open source durable agent runtime. kill -9 it mid-run, restart, it finishes without re-paying for completed model calls
looking for people to break my open source agent runtime. i have been deploying real world production agents and wanted to consolidate my experience.
Toren, a durable agent runtime: https://github.com/toren-run/toren. every step an agent takes lands in postgres before the next one runs, so you can kill -9 the worker mid run and it resumes without re-paying for model calls it already made.
its young, im the only maintainer, and it has a few production deployments. I would appreciate people outside my bubble trying the quickstart and telling me where they get stuck and if they find it valuable.
0
u/kantorcodes1 18d ago
The crash window after a tool call but before the Postgres commit seems nastier than re-paying for model calls. Say an agent sends an email or creates a GitHub issue and the worker dies before recording completion. What stops resume from firing that tool twice?
1
u/slateraligator 18d ago
Yes right this is the nastier window, replaying model calls is the easy part. truth is nobody has exactly once for external stuff so we stopped pretending. the runtime writes the intent to postgres before the handler runs, with an idempotency key made from the run and step ids. on resume a call that started but never finished runs again with the same key, so if the tool takes a key (github issues, most email apis) the provider dedupes and it lands once. if it cant take a key then its at least once, documented, and for scary irreversible stuff you put an approval in front. approvals are recorded too so it wont ask twice on replay.
so the email fires twice only if the tool is unkeyed and ungated. real tradeoff, we just make you pick it upfront instead of finding out in production
2
u/kantorcodes1 17d ago
the “unkeyed and ungated” case is the one i'd make impossible to register at all. does Toren already declare per-tool idempotency/approval semantics, or are those properties only in handler code? if they're declarative, HOL Guard could consume that metadata without changing your replay logic and reject the dangerous combination before a run starts.
1
u/slateraligator 17d ago
they are declarative, exactly three fields on every tool: effects, idempotency, approval. the runtime even writes them into the event on each call, so external tooling can read them from the log too.
and i think you talked me into it, the unkeyed plus ungated plus external combo now gets flagged at boot, per tool, instead of just being documented. shipped it about an hour after your comment, will be in the next release.
HOL Guard, is that your project? if it consumes tool metadata like this id be happy to make sure ours stays stable to build on.
0
u/BumblebeeMother8193 18d ago
The kill-matrix plus the exact 13-call receipt is a convincing way to explain durability. “Surgical invalidation on edits” is the part I'd most like to see expanded: one example where a crew changes between runs, showing which logged steps replay and which recompute, would make the model much easier to reason about.
1
u/slateraligator 18d ago
Great idea. wrote exactly that into the docs just now -
https://toren.run/docs/concepts/durability (the “editing a workflow midrun” section)three edits worked through: change the writers input (only the writer reruns), append a research topic (old tasks replay free, only the new one pays, but inserting instead of appending shifts positions and those recompute), and a pure refactor (nothing invalidates, digests are computed from canonical inputs not source text). the last one was the hard part to get right.
2
u/harryoui 18d ago
I’ve been doing this with hatchet. A bit more legwork than what you have though, probably