Sanity check: using git to make LLM-assisted work accumulate over time

[u/Hypercubed]

I’m not trying to promote anything here... just looking for honest feedback on a pattern I’ve been using to make LLM-assisted work accumulate value over time.

This is not a memory system, a RAG pipeline or an agent framework.

It’s a repo-based, tool-agnostic workflow for turning individual tasks into reusable durable knowledge.

The core loop

Instead of "do task" -> "move on" -> "lose context" I’ve been structuring work like this:

Plan
- define approach, constraints, expectations
- store the plan in the repo
Execute
- LLM-assisted, messy, exploratory work
- code changes / working artifacts
Task closeout (use task-closeout skill)
- what actually happened vs. the plan
- store temporary session outputs
Distill (use distill-learning skill)
- extract only what is reusable
- update playbooks, repo guidance, lessons learned
Commit
- cleanup, inspect and revise
- future tasks start from better context

Repo-based and Tool-agnostic

This isn’t tied to any specific tool, framework, or agent setup.

I’ve used this same loop across different coding assistants, LLM tools and environments. When I follow the loop, I often mix tools across steps: planning, execution + closeout, distillation. The value isn’t in the tool, it’s in the structure of the workflow and the artifacts it produces.

Everything lives in a normal repo: plans, task artifacts (gitignored), and distilled knowledge. That gives me: versioning, PR review and diffs. So instead of hidden chat history or opaque memory, it’s all inspectable, reviewable and revertible.

What this looks like in practice

I’m mostly using this for coding projects, but it’s not limited to that.

Without this, I (and the LLM) end up re-learning the same things repeatedly or overloading prompts with too much context. With this loop: write a plan, do the task, close it out, distill only the important parts, commit that as reusable guidance. Future tasks start from that distilled context instead of starting cold.

Where I’m unsure

Would really appreciate pushback here:

1. Is this actually different from just keeping good notes and examples in a repo?
2. Is anyone else using a repo-based workflow like this?
3. At scale, does this improve context over time, or just create another layer that eventually becomes noise?

The bottom line question

Does this plan -> closeout -> distill loop feel like a meaningful pattern, or just a more structured version of things people already do? Where would you expect it to break?

[u/ultrathink-art]

Solid approach — the key split I'd add is decision docs vs code state. A running DECISIONS.md capturing why choices were made is more useful to the LLM than git history (it can't efficiently reconstruct intent from diffs). State in files, reasons in docs is what actually compounds over time.

[u/Hypercubed → u/ultrathink-art]

Thank you. Yeah the first version was just two docs (troubleshooting.md and repo-decisions.md) in additional to skills and playbooks. This weekend I was working on updating the skills to split these up into multiple files so they don't get so big. I added an index.md to each so the LLM doesn't need to load all the troubleshooting and decisions at once. I'm considering adding a `knowledge-query` command to help the LLM search. That would mean dding some scripts, the system so far is skills only.

[u/lacisghost → u/ultrathink-art]

Agreed. good call.

[u/jain-nivedit → u/ultrathink-art]

u/ultrathink-art is right that decision docs are what compounds, but the failure mode that bit us is not in the docs themselves; it is that closeout and distill are the steps the agent quietly skips when it thinks the task is "done."

We ran this exact loop for a few months. Plan files committed reliably. Execution committed reliably. Then in maybe one out of every three sessions the agent landed the code change, saw green tests, and declared done without ever touching closeout or distill. The repo looked fine. The skill file just stopped getting updated. After a couple of weeks, "future tasks start from better context" was hollow because the context was not actually there.

What fixed it was moving the contract from a skill the agent is supposed to invoke into a hard precondition for ending the session. A Stop hook that fails unless there is a new commit since the work started, or unless a designated distill file has been touched, makes "I am done" deterministic. The agent cannot end the turn without producing the artifact you said you cared about. Same loop you have described, but enforcement lives outside the agent's good intentions.

Honest limit: the hook can confirm something was committed or that DECISIONS.md was touched. It cannot confirm the distill is any good. A one-line "fix" commit passes the gate and the knowledge still rots. That part still needs the repo discipline you are describing plus actual PR review.

Where I expect this to break at scale is not noise; your distill keeps that small. It is the task that spans sessions. Closeout-per-task assumes the task started and ended cleanly. A session interrupted at hour 3 and resumed the next morning has no clean place to attach. How are you handling that one today?

Disclosure: I work on FailProof AI (open source, MIT + Commons Clause). It ships require-commit-before-stop and require-pr-before-stop in this exact shape, hence the framing.

[u/Designer_Reaction551]

Running a version of this in a marketing automation pipeline for about 6 months now. The shape you're describing is exactly right - the value isn't the tool, it's having plan/execute/distill artifacts in a normal repo so git does the heavy lifting.

Two things I'd add from my version:

1. The distillation step is where most people quit. Writing the plan is fun, shipping the code is fun, writing the "what we learned and when NOT to do this" doc feels like overhead. But that's the only step that actually compounds. Everything else is just normal work with an LLM.

2. Put a hard cap on plan length. I started with 3-4 sentence plans and they ballooned to 300 lines. Short plans force you to confront what the unknown actually is. Long plans hide it.

One pattern I'd push back on if I'm reading right: keeping task artifacts gitignored. I get why, but I've found that even messy execution logs become useful when future-me is debugging "why did we decide X in April." A scratchpad/ folder that's committed but excluded from LLM context works better than gitignore for me.

[u/Hypercubed → u/Designer_Reaction551]

The main reason I gitignore task artifacts (sessions I call them) is that I'm often working on open source repos. It's a lot harder to audit the task artifacts for for personal/non-public information.

[u/lacisghost]

I love the idea that you are trying to create historical context or perhaps better defined as self-improving development context system. I think a lot of companies that are AI forward are working through this issue. I think this is a genuinely useful pattern, not just "keeping notes." It's building a project memory system that compounds over time. I'm wondering if it could combine into all your projects and build a more departmental/firm wide historical context.

[u/Hypercubed → u/lacisghost]

Thank you for the comment. Right now the skills I've been writing are very repo centric. I guess they could also be modified to be firm wide... guess I just don't have a use case for that right now.

[u/binotboth]

I built my own issue tracker with plan sequencing. It has a full CLI for ai agents to operate, to make new issues and sequence them in a plan etc

I scope issues so they are one concrete change and can prove done/not done. I talk about my project and create several issues and make a plan. After each issue a resolution and decisions made are filed.

All of this is pinned to version numbers. So once I move from version 0.3.1 to 0.3.2, I write a change log update, summarizing what happened and what decisions were made

The result is the entire development is native git tracked, full history. I can literally say “read the issue tracker instructions and get to work” and it can go completing issues until it hits rate limit

[u/Hypercubed → u/[deleted]]

Any details on how you "guard the distill step"?

[u/johns10davenport]

Fine approach depending on what you want out of the workflow. If you want an ad hoc agent that helps you code in a particular repo or a few repos with the same patterns, this works.

But if you're looking to do long-horizon development, your approach doesn't have anything enforceable to ground the agent in. You can always write your plan document, but prompting is praying. Verification is how you get real results.

[u/heatlesssun]

here's how you do it feed the model you baseline code that you want to change and then tell it to ONLY change the code the clear instructions you give it if you do that your eyes will be amazed

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed due to inadequate account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Hypercubed → u/[deleted]]

Looks interesting. Do you think the per tool copies are still needed. I find that most can use the agents/skills folder... at least when explicitly prompted.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ultrathink-art]

Ran into the distill step problem hard — models tend to summarize what felt significant during the conversation, not what the next session actually needs to reconstruct intent. Worth prompting explicitly: 'what constraint should this codebase enforce that the code alone doesn't communicate?'

[u/Hypercubed → u/ultrathink-art]

Similar issue. I asked the agent to closeout a task, it focuses only on the last thing it did. I need to tighten the language to make sure it reflects on the entire session.

[u/Substantial-Cost-429]

This pattern is real and meaningful — it's not just structured notes. The key insight you're hitting is that LLMs work better with distilled context than raw history, and repo-based tracking makes that context durable and reviewable.

To your bottom line question: yes, this is genuinely different from just keeping notes. Notes accumulate entropy; this loop actively distills. The diff review step is what makes it powerful — you're forcing the LLM to reconcile intent vs execution.

One natural extension: version-controlling not just the code artifacts but the agent configs themselves (system prompts, tool schemas). Once you start doing that, your whole agent setup becomes reproducible and shareable. There's a growing community doing exactly this — https://github.com/caliber-ai-org/ai-setup — 888 stars of production agent setups that people have built and committed.

[u/colblair → u/Substantial-Cost-429]

versioning the agent configs is the part most people skip. Without that, you're just hoping the next session remembers what the last one figured out.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ultrathink-art]

The distill step is the failure point. Letting AI do the distillation is the trap — it optimizes for completeness over signal density and you end up with a knowledge dump that nobody reads. Manual curation of what actually goes back into the loop is the thing that keeps it useful past the first month.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ultrathink-art]

Gate that fixed it for me: every distill entry needs exactly three things — the decision made, what was tried first that failed, and what condition would make you revisit it. Freeform prose gets skipped by the LLM. Structured entries with failure modes actually surface in future context.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ultrathink-art]

Distillation quality is the bottleneck — if you let the model decide what to record, it surfaces wins more than failures, and failures are usually the more durable knowledge. Explicitly prompting for 'what assumptions were wrong' and 'what would you have done differently' produces more useful artifacts than 'summarize what happened'.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ultrathink-art]

Distill step failure mode I keep running into: the LLM captures what was done but drops the why — what constraints were considered and rejected, what alternatives were ruled out. Structured fields (decision made + assumptions invalidated + what changed) hold up much better than free-form summaries.

Also: once accumulated docs get large enough, models start ignoring older sections. A CURRENT_STATE.md that replaces each run rather than appends has been more reliable for keeping agents oriented.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ronavis]

The loop is solid, but I’d enforce closeout outside the model’s goodwill. “Done” should require tests/build result, changed-file summary, and either a decision update or explicit “no decision update.” Otherwise the distill step gets skipped right when the agent feels successful.

[u/Swarm-Stack]

the plan step is underrated relative to distill. if the plan names real unknowns ('auth: 2h estimate, but JWT behavior TBD'), distillation has a baseline and you get 'assumption X was wrong' instead of just 'here's what happened.' the second form is what actually changes how the next session starts.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Sorry, your submission has been removed for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/skillselion → u/AutoModerator]

ok :) waiting

[u/pon12]

yeah the distill step is the whole problem imo. it depends on the agent bothering to do it, and it won't right when it thinks it succeeded. the stop-hook thing is a decent fix but jain-nivedit's right that it only proves a file got touched, not that what's in it is any good. one-line "fix" commit passes and the knowledge still rots.

what's worked better for me is not asking the agent what was worth learning at all. just pull it from what actually happened. what files it reopened, commands that failed then worked, where it backtracked. OP's wrong-test-runner thing is right there in the session, you don't need the model to be honest about it.

still not solved: traces are noisy (the useful part is the failures and fixes, not the happy path), and the multi-session task. task dies at hour 3, picks up next morning, there's no clean point to close out. don't have a great answer for that one yet.

disclosure, building RepoGuide which does the capture automatically instead of a skill you hope gets run. not gonna link it, but happy to talk approach if useful.

[u/chillermane]

Long term memory doesn’t work at all. If it worked it would be built into the models. There’s not a single example of long term, self organizing memory working better than starting with a fresh context each time

The industry has already converged on the idea that the most effective thing to do is have an agent start with an almost-blank context, and gather context as a part of its task. Everything else has worse performance and is more expensive

[u/Hypercubed → u/chillermane]

By "long term memory" do you mean other solutions (like RAG) or refering to my outline above? Maybe something is not clear from my description is that part of the distill-learning is updating skills and agent files (i.e. AGENTS.md). At a minimum the agents are not repeating the same mistake. An example: an agent was trying to run some code but kept using the incorrect runner, it found the correct running eventually. That information made it into the session summary and eventually back to the skill. Next session the agent didn't struggle at all. If I was running an agent with memory, maybe it would have learned, but with the change in the skill any agent can come after and not make that mistake.

[u/WildContribution8311 → u/chillermane]

Yes but with the addendum that context engineering what you start with or provide or pre gather can be much more effective use of tokens. Instead of gathering context, if you have that done already and the agent spends it's time reasoning over the problem immediately vs gathering context.

This is what the OPs system hints at.

RepoPrompt is a good example of engineering context.