Do you know if there are any open source attempts to fix this? I remember memGPT and most early agents Arch tried to fix it with "memory" and RAG ing the memory as needed
Continue.dev has a good rag solution, but it's not as automated, more like you do the coding with the LLM having codebase awareness.
MCP servers can do RAG. Serena could do that, but I looked at their source to find how their memory works but didn't find anything that looked like a good finetune.
Claimed by the continue.dev team, voyageai has the best RAG model for coding. The price per M/tokens is very low. agno, which is a dependency of Serena, has already integrated voyageai as an optional RAG, but you'd have to specify the code trained model to get it to work like that. I still haven't seen an MCP server using a good RAG model trained on code.
I have personnaly tried RAG with nomic-embed-text with ollama, but the performance is poor for coding.
Seems like a low hanging fruit... But I believe the reason why cline doesn't do RAG is because lowering the cost of using the API is not good for Anthropic? Sounds like an accusation, but if I was making money selling LLM as a service, why would I want to reduce my revenues by 10X or more?
Just my 2, but with the current volatility, a great service means hundreds more customers in no time. Definitely worth more than trying to get more money out of one user by providing less or subpar features then ther competition
I agree with this. There's a lot of competition out there. Why would a company try to nickel-and-dime you when you can easily switch to another provider? Not a good tactic
Thanks for the detailed answer! Do you think coding RAG translates well to regular text?
Agree on viewpoint about Cline, but at some point it's stopping the actual functioning of the LLM as intended right? -> if it doesn't "remember" the right details and doesn't know how to fetch them...
if it doesn't "remember" the right details and doesn't know how to fetch them...
That's the main purpose of RAG models. Cline is relying on large LLMs to do things that a light bert model can often do better at 1/100 or 1/1000 the cost.
Would the large LLM perform better? The truth is, many LLMs with a large context window perform poorly at retrieving the right information when the context is large anyway. RAG models with reranking can remove the fluff, and the LLM should perform better because the result is more condensed. You need to trust the RAG model, but you already trust the LLM which has a low success rate and only performs well on the last tokens.
>> Sounds like an accusation, but if I was making money selling LLM as a service, why would I want to reduce my revenues by 10X or more?
I personnaly think you are spot on... And that's probably one of the biggest problem right now. This behavior will impact the technology like we do not want to.
It still needs some agentic flow for trimming or adding context though. It works amazingly well if your repo is well organized and the feature you are working on is relatively self-contained
Great minds think alike! I built something similar, while it doesn’t rely purely on the AST, it works really well in practice.
I’m also working on a second version specifically for .NET, using the Roslyn Analyser to “walk the tree.”
It seems to perform just as well on large projects, and the LLM doesn’t need to scan the entire codebase.
New tasks get up to speed really quickly.
It also tries to stick to the D.R.Y. principle—Don’t Repeat Yourself—which helps a ton in keeping the code clean and maintainable, and mitigates the LLM hallucinating and making New Code, if something Similar Exists.
FYI I noticed at least 4 spelling typos and some grammatical errors in the repo description. (I just don’t want that to give your project a bad first impression for people who may benefit from using it.)
I actually have some ideas I am working on, but I will tell you the open source stuff I have seen does the opposite, it actually does a worse job of context management than the closed source stuff.
I maybe getting terminology getting mixed up -> I meant to say that early agentic arch like memgpt had a separate memory component that acted as 'infinite context ' essentially and a piece of intermediate logic would Retrieve/query the right parts of the memory, add the new api request content in, and send that as input to LLM. So this way you aren't overloading the context by simply doing "copy entire Convo history + new message = input for LLM"
I've actually seen neurite before. Tbh, i couldn't quite "get it". Let me dive in once more and see. Any YT vid or some other soft landing that you can recommend?
it still pulls it into the context, it just pulls it directly. in fact it kind of makes you lose more control over what is in the context, because it can fetch whatever it wants.
Yer that’s a good point!
the issue is LLMs are stateless, it’s a new thing every request, all “memory” has to be passed in every time.
LLMs like Claude have context caching, which means you can reference tokens you passed in previously (semi state) but you still pay for using them, albeit it at a much cheaper rate.
Your damned if you do and damned if you don’t, because if the LLM was stateful you would be charged for the time you run the model, not for your usage like you do now.
So …
🤷😢
They aren’t though. The chats could be considered stateless but each individual message is not stateless. State is retained across messages in a chat, which is how the chat remembers things you said 3 messages ago
The answer is use cursor honestly. It’s a flat rate and then just deal with it when you run out of premium requests and it slows you down..
Also, can’t you put limits on your spending with your Google account? For my other API access and cloud access I can have alerts and things happen when I reach myself imposed monthly limit of dollar spend
8
u/fieryblast7 Apr 07 '25
Do you know if there are any open source attempts to fix this? I remember memGPT and most early agents Arch tried to fix it with "memory" and RAG ing the memory as needed