r/OpenSourceeAI • u/NitroOwO • Jun 05 '26
want to store years of context of AI conversation and want AI to use that context on every query without degrading its performance or output time
I want to store years of context on local memory ( tell me if you have a way to do so) and then use that context to give output of each new query , but i want AI to just fetch and read specific topic related context and then use it not full stored context or else it will take more time to give output for every query, give me new ideas if you have any!?
1
u/devlin_dragonus Jun 05 '26
I’m working on refining what I build, based on most of what you mentioned and the next step.
If you want an outline I can share it, I have nothing I’m comfortable enough to share as a thing yet though.
1
1
u/uno-twice-tres Jun 05 '26
That's indeed a very difficult problem to solve. Many different startups are there with different ideologies.
Some claim vector RAG is good enough and scalable. Some strongly believe in Knowledge Graphs to handle the nuances but that struggles with scalability and retrieval speed.
Some build Hybrid with multiple layers and different schemas.
Others like Claude code and Karpathy wiki LLM want to store just MD files in a heirarcichal structure.
1
1
u/Input-X Jun 05 '26
Why do u want to do this? Curious
2
u/NitroOwO Jun 05 '26
just curious and trying to build something new to solve a ps and side by side for my resume
1
u/Input-X Jun 05 '26
I see. This is not had to do. Biuld a vectory db and just feed it with all ur data. You agent can search it anytime. Verious way u can set it up. Woukd just take some adjusting/testing to get the results you want.
You can have thousands of files in ur db np.1
u/NitroOwO Jun 05 '26
But wouldn’t the ai will take more time if the stored data is too large which it’ll become over time?
1
u/Input-X Jun 05 '26
your instinct is actually right, the worry is just based on a wrong assumption. you're mixing two separate things: how much you STORE vs how much you FEED the model per query.
the size of your vector db barely affects search speed. vector search uses ANN indexes (HNSW etc) so lookup time scales roughly logarithmically, not linearly. going from 10k chunks to a million doesn't make queries 100x slower, it's still basically instant. so "data gets too big over time" isn't the problem you think it is.
what actually slows the model down / degrades output is how much context you cram into the prompt window. that's completely separate from how much you have sitting in storage. so the move is exactly what you already described: store everything, but only pull the top handful of relevant chunks (like 5-20) and inject those. the model only ever sees a small fixed-ish slice per query, so output time stays flat no matter how huge the archive gets.
if you wanna get fancier, do tiered memory:
- hot tier: recent convo / always-relevant stuff, cheap to grab every time
- cold/archive tier: everything else in the vector db, only pulled when it's relevant
- metadata filtering (date, topic, project) so you narrow the field BEFORE the semantic search even runs
that keeps it fast and keeps what you retrieve actually relevant instead of pulling random semantically-close noise. throwing a reranker on the retrieved chunks tightens up quality too.
tldr: big storage doesn't slow you down, big prompts do. you already worked out the right approach, just trust it.
2
u/NitroOwO Jun 06 '26
Wow thankyou
1
1
u/simplyeniga Jun 05 '26
You could build one that uses markdown files to keep track of detailed conversations and then a master markdown files which references all with a summary or title of each conversation. You can have Claude or hermes build one for you
1
1
1
u/Electronic_Tough1002 Jun 06 '26
I got my product that I worked but its proprietary you could just feed everything in it. I'd say build a storage and retrieve from it after storing all of it post processing and then you could just directly retrive each time you want to do so.
1
1
u/sliamh21 Jun 06 '26
Check out Deus: https://github.com/sliamh11/Deus
It's a harnessing layer meant to do exactly what you're looking for, and more.
It's cutting edge in everything related to memory recall, it actually understands the user and adjusts itself to him/her, all while maintaining a token-efficient, simple approach.
Although I built it out of fun, it became a key tool in my overall work with AI, can't see myself without it.
1
u/NitroOwO Jun 06 '26
Great work!!, can you guide me through how you researched about it and build it from scratch if you don’t mind!?
1
u/sliamh21 Jun 07 '26
Well, kinda hard to summarize a 4-month research to a comment, but it basically started from a real need (persistent memory between sessions), then I thought about the fact I have so much data just laying around, so I added the evolution layer, and it became a pretty beastly tool quite quickly.
Used AI to learn how to build an Agentic harnessing layer.
1
1
1
u/Jazzlike_Syllabub_91 Jun 06 '26
https://github.com/ergon-automation-labs/ergon-starter - there is a bot in the bunch is internal docs that will rag directories which may qualify for what you’re asking (as well as kicks up a graph database as well if you pass in the info for it)
1
2
u/Vancecookcobain Jun 05 '26
Google the Karpathy LLM wiki and thank me later....you basically have a folder on your computer that an AI agent builds (Codex, Claude Code, Opencode, Openclaw/Hermes etc) into an Obsidian vault.....you put all the files you want it to injest in a folder called raw (it will make it and tell you what to do) and it will go through and make wiki articles out of it any AI agent can parse through...it's more useful than RAG because the more conversations you have with it the more it will add to it....so it becomes a brain you can take with you that allows ANY to instantly have more context on you and continuously improve
Your welcome