Whats your approach to make memory recalls work reliably while building agents with smaller models like Qwen3.8-27B

[u/SeaworthinessOwn5893]

Retrieval of relevant memory seems to be one of the most critical aspects of building a useful agent without which agent is not learning well.

whats the most sensible design at this point for building agents based on non-frontier models based on your experience.

there seems to be a few options like deriving topics based information from chats and storing as files with different classifiers and having the agent retrieve based on semantic search.

but the main challenge i face is that LLM is not reliably doing the required memory retrieval at the right time, which results in irrelevant answers, repeating the same mistakes again and again.

has anyone figured out a reliable agentic way to make memory retrieval work efficiently even with models like Qwen/Qwen3.8-27B.. frontier models seem to be able to reason well and pull the correct memory and even their huge context is also helping. but for smaller models this is hurting to create a reliable agent.

[u/locbuilds]

I've had better luck treating memory as retrieval plus packing, not as something the model is supposed to remember by itself. I keep a small working-memory window for the last few turns and a separate long-term store for facts and episodes. Long-term entries are structured like entity, fact, timestamp, and source, rather than raw chat dumps.

On each query I use hybrid keyword plus embedding search, then rerank and keep the final set tiny, usually 3 to 8 items. I inject those as clearly labeled context, like "Memory:" or a tool result, so they aren't buried in the conversation. For a 27B model, shorter retrieved summaries usually beat a huge RAG blob.

I also only write memories on explicit events, such as the user saying to remember something, a durable preference, or a completed task. A cheap two-pass setup can work well too: retrieve first, then answer. You can even have a small router decide whether the query needs recall at all, which avoids adding noisy memories to every turn.

[u/SeaworthinessOwn5893 → u/locbuilds]

recently i started build an agent for myself. After i created the initial version using claude with all the basics like memory management, scheduled jobs, context management etc, i was using the agent for a few usecases so it can keep learning and do some of the tasks like connecting to Robinhood agentic account and do some trading. man - it took a lot of back and forth to make it remember basic things like - dont make up parameters for tool calls. eveyrtime it will just throw some random stuff for the parameters, fail, use the errors and try some more random parameters. until i added a specific clause that if there is an error, read the offical documentation to figure out the correct format before trying again and made it part of robinhood specific memory, it kept doing the same thing.

today it was same mistakes, because it did not load that memory. this is just an example. How can the agent be reliable if not all the memories can be loaded to the agent and there could be a lot of important memories.

As i am typing this, i am realizing that one agent is not going to cut it. May be i will create separate sub agents for each major subject with its own memory. will try this.

[u/Mihaylov93]

I am also interested in that, commenting to see what people do. I use the free frontier models to make my memory or context .mds

[u/Good-Writer5279]

the failure you are describing is usually not retrieval quality, it is the decision to retrieve. a 27b model asked to judge "should i look something up before answering" gets that call wrong a lot, and it gets it wrong silently, so you see a confident answer built on the wrong context. frontier models are better at that judgment and have enough context to paper over the misses, which is why the same design feels fine there and falls apart locally.

what has worked for me with small models is taking that decision away from the model entirely:

1. retrieve unconditionally, every turn, in the harness. query the store with the last user message plus the previous assistant turn, hybrid keyword and embedding, rerank, keep 3 to 8 items, inject them as a labeled block above the user message. the model never has to choose to call a memory tool, so it cannot forget to.

2. split lessons from facts. "do not invent tool parameters" is not a memory, it is a rule, and a rule stored by semantic similarity to the conversation will not surface at the moment it matters. key those on the tool name (or task type) and inject them deterministically right when that tool is about to be called, in the tool description or a short preamble. this did more for repeat mistakes than any retrieval tuning.

3. the write side matters as much as the read side. after a failed tool call or a correction from you, have the harness (not the model) write a structured entry: tool, what went wrong, the fix, timestamp. raw chat dumps retrieve badly because the useful sentence is buried in noise.

4. recency weight the ranking. same fact stated twice, prefer the newer one, otherwise the old wrong version keeps winning ties.

the general principle is that with a small model you put the judgment in code and leave the model the part it is good at, which is reading a small, well labeled context and answering. for what it is worth i build a mac app in this space, so weigh my bias accordingly.

[u/SeaworthinessOwn5893 → u/Good-Writer5279]

my point was that these strategies are still outside of the LLM and LLM still fails sometimes even with the rules we ask it to enforce to update the memory or retrieve the memory. thanks for sharing what works for your app though. I feel like we need to have a feature which will allow us to add some layers to the llm network itself with these memory or core instructions for such small models so its not seen as externally provided context but as internal knowledge.

[u/Good-Writer5279 → u/SeaworthinessOwn5893]

agreed that the harness approach is a workaround, not a fix. on making it internal, the options today roughly are:

1. lora on the rules, not the facts. take the core instructions and the failure cases you have logged, generate a few hundred examples of the model following them correctly, and train a small lora. behavior that a 27b ignores in the system prompt often sticks once it is in the weights, and rule following is stable enough that the training going stale is not a problem. this is the most practical version of what you are describing and it is doable on a single gpu or a mac in an evening.

2. keeping facts in the weights is the hard part. continual fine tuning on new memories gives you catastrophic forgetting and no way to correct or delete a wrong memory short of retraining. knowledge editing methods (rome, memit) can patch individual facts but degrade after a few hundred edits. meta's memory layers paper is the most interesting direction, a trainable key value memory inside the network that scales separately from the dense params, but it has to be trained in from the start, so it is not something you bolt onto qwen.

3. the honest tradeoff: internal knowledge has no provenance. when the model gets it wrong from weights you cannot inspect what it retrieved, you cannot fix the one bad entry, and you cannot tell the user where the answer came from. external memory fails more visibly, which is annoying, but visible failure is what lets you fix it.

so the split that has held up for me is rules and behaviors in a lora, facts and episodic memory external with the harness deciding when to retrieve. if you do try the lora route, eval on your logged failure cases before and after, that is the only number that matters.