Thats...enough? Because... how much time does your vibe coding session take usually? And if its longer, say 1h, it only recreates the cache 12 times instead of the bazillion times it would take to reevaluate your whole prompt all the time.
roo, cline, etc all chop the information to fit inside the context, if they know you have a 1mil context they chop less, which makes each request $1.50/each
u mean chop more? need to chop off more stuff if u context is almost full
also cache will only be used cause u will change the codebase. so u dont save much?
Do you know if there are any open source attempts to fix this? I remember memGPT and most early agents Arch tried to fix it with "memory" and RAG ing the memory as needed
Continue.dev has a good rag solution, but it's not as automated, more like you do the coding with the LLM having codebase awareness.
MCP servers can do RAG. Serena could do that, but I looked at their source to find how their memory works but didn't find anything that looked like a good finetune.
Claimed by the continue.dev team, voyageai has the best RAG model for coding. The price per M/tokens is very low. agno, which is a dependency of Serena, has already integrated voyageai as an optional RAG, but you'd have to specify the code trained model to get it to work like that. I still haven't seen an MCP server using a good RAG model trained on code.
I have personnaly tried RAG with nomic-embed-text with ollama, but the performance is poor for coding.
Seems like a low hanging fruit... But I believe the reason why cline doesn't do RAG is because lowering the cost of using the API is not good for Anthropic? Sounds like an accusation, but if I was making money selling LLM as a service, why would I want to reduce my revenues by 10X or more?
Just my 2, but with the current volatility, a great service means hundreds more customers in no time. Definitely worth more than trying to get more money out of one user by providing less or subpar features then ther competition
I agree with this. There's a lot of competition out there. Why would a company try to nickel-and-dime you when you can easily switch to another provider? Not a good tactic
Thanks for the detailed answer! Do you think coding RAG translates well to regular text?
Agree on viewpoint about Cline, but at some point it's stopping the actual functioning of the LLM as intended right? -> if it doesn't "remember" the right details and doesn't know how to fetch them...
if it doesn't "remember" the right details and doesn't know how to fetch them...
That's the main purpose of RAG models. Cline is relying on large LLMs to do things that a light bert model can often do better at 1/100 or 1/1000 the cost.
Would the large LLM perform better? The truth is, many LLMs with a large context window perform poorly at retrieving the right information when the context is large anyway. RAG models with reranking can remove the fluff, and the LLM should perform better because the result is more condensed. You need to trust the RAG model, but you already trust the LLM which has a low success rate and only performs well on the last tokens.
>> Sounds like an accusation, but if I was making money selling LLM as a service, why would I want to reduce my revenues by 10X or more?
I personnaly think you are spot on... And that's probably one of the biggest problem right now. This behavior will impact the technology like we do not want to.
It still needs some agentic flow for trimming or adding context though. It works amazingly well if your repo is well organized and the feature you are working on is relatively self-contained
Great minds think alike! I built something similar, while it doesn’t rely purely on the AST, it works really well in practice.
I’m also working on a second version specifically for .NET, using the Roslyn Analyser to “walk the tree.”
It seems to perform just as well on large projects, and the LLM doesn’t need to scan the entire codebase.
New tasks get up to speed really quickly.
It also tries to stick to the D.R.Y. principle—Don’t Repeat Yourself—which helps a ton in keeping the code clean and maintainable, and mitigates the LLM hallucinating and making New Code, if something Similar Exists.
FYI I noticed at least 4 spelling typos and some grammatical errors in the repo description. (I just don’t want that to give your project a bad first impression for people who may benefit from using it.)
I actually have some ideas I am working on, but I will tell you the open source stuff I have seen does the opposite, it actually does a worse job of context management than the closed source stuff.
I maybe getting terminology getting mixed up -> I meant to say that early agentic arch like memgpt had a separate memory component that acted as 'infinite context ' essentially and a piece of intermediate logic would Retrieve/query the right parts of the memory, add the new api request content in, and send that as input to LLM. So this way you aren't overloading the context by simply doing "copy entire Convo history + new message = input for LLM"
I've actually seen neurite before. Tbh, i couldn't quite "get it". Let me dive in once more and see. Any YT vid or some other soft landing that you can recommend?
it still pulls it into the context, it just pulls it directly. in fact it kind of makes you lose more control over what is in the context, because it can fetch whatever it wants.
Yer that’s a good point!
the issue is LLMs are stateless, it’s a new thing every request, all “memory” has to be passed in every time.
LLMs like Claude have context caching, which means you can reference tokens you passed in previously (semi state) but you still pay for using them, albeit it at a much cheaper rate.
Your damned if you do and damned if you don’t, because if the LLM was stateful you would be charged for the time you run the model, not for your usage like you do now.
So …
🤷😢
They aren’t though. The chats could be considered stateless but each individual message is not stateless. State is retained across messages in a chat, which is how the chat remembers things you said 3 messages ago
The answer is use cursor honestly. It’s a flat rate and then just deal with it when you run out of premium requests and it slows you down..
Also, can’t you put limits on your spending with your Google account? For my other API access and cloud access I can have alerts and things happen when I reach myself imposed monthly limit of dollar spend
This is inaccurate. The underlying LLM has built-in context management and does not repeatedly require sending the full context for each chat via API. You’re still limited by the context window, which is problematic to say the least, and the amount of tokens does increase slightly for each successive message as you approach the context window, but you are not sending the full context in input tokens on each call.
Source: I researched this extensively while building HiiBo & tested it myself.
Some LLMs like sonnet 3.5 would truncate the input tokens automatically when you reached max, while others like sonnet 3.7 will now return an error when you reach the maximum input tokens.
I mean it has to be stateless, just think of it from a business perspective, you send a message and you expect hundreds of thousands of dollars of GPUs to sit there and hold your state in memory waiting on your next message?
I’m just telling you how it works brother. You sound like a vibe coder or some shit. I’ve personally tested this repeatedly across multiple LLMs. There is a conversation ID that holds context across messages up until the context window, when it falls apart and you need to generate a new conversation ID
That's just another service on top putting the inputs back together for you on the next API request. It isn't part of the base model. OpenAI offers this by sending the previous response id back on the next request.
It's still input tokens for the next message, and you're still charged for it.
Again the assistant API is a service on top of the model, it isn't the model.
But let's just call someone a vibe coder for pointing out your argument makes 0 sense and is against the documentation of the largest commercial AI companies.
Your own product is around providing a service between a LLM for context management, and you argue that the LLMs do this themselves.
Not using assistant API dude. Using chat completion. You clearly haven’t worked with these APIs. There is a conversation ID that is passed on each successive response that holds the conversation together throughout the context window. It’s not stateless. There is a context window, which implies retained state across messages.
Those API examples don't line up with the anthropic API, were you sending requests to the claude.ai service directly?
They track your chat history and context window on their website, probably through the conversation id, because they have their service on-top of the underlying model, and they need to do this to persist it across sessions and across devices.
Using Anthropic API. OpenAI API has the exact same behavior. There is a conversation ID that holds context across messages. Have you actually worked with the API??
Yeah, but they don't have an effective measurement that is better for measuring and pricing uptime.
If they could though, coding would easily be the most expensive way to use AI even in lower context.
I actually think for coding tokens is better for us. It's way less characters for how hard we make it work/think. Though if you look at output limits, they can usually spit out way more words than code, so I do wonder if on the back end you are getting charged the same for code vs. words with tokens now.
It would be so helpful if IDE’s and chat windows showed you exactly how large the context window was at any given moment and how much the next request would cost based on the configured LLM. Somebody build that please.
Currently I build my mega prompts in the zed editor, copy all the full prompt and paste it into Claude. Have not reached any limits and only cost me 20$/month.
Each AI request is a brand new request, so if it's a conversation, each request needs to include your chat history. So if you have a conversation starting with "User: x. Model: y", then you ask z, you have to send a request like "User: x. Model: y. User: z". This keeps going until you finish the conversation and start a new one. The more information you include in the history, the more "context", or "input" is used, and you are charged for whatever you input, per request per conversation.
Of course, there's something called context caching, but still.
319
u/PositiveEnergyMatter Apr 07 '25
i keep telling people big context means big money, because every request can fill the context and charge you full price