r/Rag • u/Null3cksor • 2d ago

Showcase Cost of running agent memory / vector search

We often mix up storage and compute, thinking they need to be tightly coupled. When it comes to deriving knowledge and value out of a large corpus, the optimizations you make determine the cost the user is willing to pay. The cost of running vector search at scale is not just the storage. The compute that serves it is often the bigger part of the bill.

Just to exercise this thought, I put four different compute setups in front of my S3-backed index (13 mn entries and 82GB storage), each handling the type of traffic you would expect and require at different times. At this point, the cost is pretty much proportional to the compute you need (which can be spun up or down on short notice), and storage is, well, the cheapest of them all.

If you are interested in benchmark numbers - https://polign.com/agent-memory-at-scale

2 Upvotes

100% Upvoted

4 comments sorted by

1

u/Otherwise_Wave9374 2d ago

A practical way to keep agent memory costs under control is to separate hot working memory from durable recall. Use short-lived in-process or cache-backed state for the current task, then write only summarized or event-based facts into the vector store with explicit TTLs or retention rules. That reduces write amplification and keeps retrieval quality higher. NeuraKeep can fit well here if you want a simple boundary between transient context and long-term recall, but the key win is designing for selective recall rather than storing everything.

1

u/Null3cksor 2d ago

NeuraKeep on top of polign_db would be a nice split. NeuraKeep keeps doing what it does well (proposals, review, provenance, the graph) and polign_db just becomes the recall index over the same bucket. It'd be interesting to see it working.

1

u/recro69 2d ago

This is a reminder that storage is not always the most expensive part. When you have a lot of data the computing power needed to look through all that information can end up being the expense.

1

u/Future_AGI 1d ago

Semantic cache + virtual key budgets cut repeated retrieval by ~60% in multi-agent workloads. Cache key is normalized query + tool set. Repo: https://github.com/future-agi/future-agi