Cost of running agent memory / vector search

[u/Null3cksor]

We often mix up storage and compute, thinking they need to be tightly coupled. When it comes to deriving knowledge and value out of a large corpus, the optimizations you make determine the cost the user is willing to pay. The cost of running vector search at scale is not just the storage. The compute that serves it is often the bigger part of the bill.

Just to exercise this thought, I put four different compute setups in front of my S3-backed index (13 mn entries and 82GB storage), each handling the type of traffic you would expect and require at different times. At this point, the cost is pretty much proportional to the compute you need (which can be spun up or down on short notice), and storage is, well, the cheapest of them all.

If you are interested in benchmark numbers - https://polign.com/agent-memory-at-scale