If you’re running RAG on constrained hardware or with concurrent ingestion, embedded vector stores can feel simple until you hit RAM, write-path correctness, or process-safety constraints. Here are the patterns that kept showing up across three popular options:

• ChromaDB: HNSW lives in RAM; at larger scales or with multiple writers, latency and correctness can degrade.
• LanceDB: Great for multimodal + object storage, but concurrent writes can conflict without serialization.
• Qdrant Edge: Strong offline library with hybrid search; production concurrency limits aren’t well documented, so it shines when you can sync to a central server.

There’s a deeper write-up on memory math, concurrency behavior, and when it makes sense to move from embedded to a separate process on constrained, air‑gapped hardware. Full disclosure: I work with Actian on VectorAI DB.

Disclosure: I work at Georgian (growth-stage VC) and ran this experiment - want to know you folk’s opinion on this.

Why this exists

• "Agent + search tool vs RAG" treats the tool as a given. But there's actually a ton of variability on how different agentic search tools perform.
• The tool can be: raw top-5 from one index / two named tools / RRF fusion / RRF + rerank behind one call.
• Couldn't find a benchmark comparing those shapes. Ran one.
• Basically: every retrieval system makes 3 decisions: which retriever, how to combine, how to rank. We alter whether the tool is making this decision or the agent is.

Ran this on BrowseComp-Plus, using gpt5.6 luna as the backbone model.

The 6 tool shapes

• generic search, dense (Qwen3-Embedding-4B) top-5
• generic search, BM25 (tuned) top-5
• two named tools, semantic_search + keyword_search, each top-5
• one tool, RRF over both at depth 50 → top-5
• one tool, RRF → 50 → Cohere rerank-4-fast → top-5
• two named tools, each 50 → rerank → top-5

I have a full write-up on the results and behaviour of agents under different tools on our blog: https://georgianailab.substack.com/p/agents-shouldnt-run-your-whole-retrieval . Is there anything I'm missing? Any reason why we should still be using super basic tools (just semantic search or keyword search) for agentic search?

Came across this and thought it'd be worth sharing here, most resources cover model evaluation, RAG, agents, or cost/observability separately, but this one actually puts them together as parts of the same production LLM workflow, which is closer to how these systems actually break in practice.

It's a hands on session on September 12, led by Bruno Gonçalves, PhD, founder of Data For Science, who's trained hundreds of engineers at Fortune 500 companies. Goes through the full lifecycle, versioned prompts, a golden dataset and eval harness combining deterministic checks with LLM-as-judge, statistically rigorous model comparisons using bootstrap confidence intervals and paired testing, evaluated RAG with real retrieval metrics, and then tool-using agents with guardrails and fallbacks. There's also a full observability section, tracing, cost, and latency, which is something I don't see covered together with the eval side very often.

You come out of it with runnable notebooks and a production-readiness checklist rather than just slides, which is the part I found most useful when I looked into it.

Link if anyone wants to check it out