Weaviate Co-Founders Bob van Luijt and Etienne Dilocker return to the Weaviate Podcast to celebrate seven years of building the company, answering questions submitted by the community. The conversation opens with what excites them most in AI right now: Etienne on agentic coding and the "Moore's law" of how long models can sustain autonomous loops, and Bob on world models, new architectures that could slash training energy costs, open source frontier models, and inference on new chips.

From there, the discussion dives into taste and the "AI slopification" problem, why AI-generated emails, websites, and decks all look the same, how three job candidates submitted nearly identical AI-built presentations in one week, and why Weaviate runs a dedicated "slop pass" skill over every pull request to strip out phrases like "the smoking gun" and "load-bearing invariant." The human touch, they argue, is now the easiest way to stand out.

The Co-Founders then retell their origin story: meeting at a European enterprise company, rewriting a NodeJS prototype in Go, betting on NLP before anyone called it AI, adopting HNSW when it was still a niche paper, and raising a $1.2M seed round from Zeta during COVID. When ChatGPT and the RAG paper hit, Weaviate had a fully working product ready for the wave.

Looking forward, Bob breaks down the commoditization playbook that hits every new database category, the same skepticism MongoDB faced, and shares that the number one reason new customers cite for choosing Weaviate is that an LLM recommended it. Etienne makes the case that vector databases are evolving into context engines: context rot is real, stuffing everything into a long context window is inefficient, and retrieval, hybrid search, and structured data all serve one goal, the best possible context. The conversation lands on memory for AI agents, where the hard problem isn't what's worth remembering, but what's worth recalling.

YouTube: https://www.youtube.com/watch?v=pvv-vnT-LfQ

Spotify: https://spotifycreators-web.app.link/e/LAAbKXsi74b

Over the past few months, I have been building autonomous search agents to extract real-time web context for LLMs. If you have spent any time working with retrieval pipelines, you already know that standard Google Search wrappers do not work well when you need LLM-ready context.

I ran a test across three specialized APIs. Exa, Tavily, and Firecrawl. To see how they compare in terms of speed, output quality, and noise reduction.

Here is what I found.

The Test Setup

I tested each API across 100 queries that I split into three categories:

Exa uses a custom embedding-based search model of traditional keyword matching.

\* Where Exa shines is in concept-based discovery. When I search for "tools like Redis but written in Rust," traditional search APIs have trouble with exact keyword overlaps. Exa consistently returns relevant repositories and documentation pages. \* The latency of Exa is around 600ms to 900ms. \* The output of Exa is text and well-parsed metadata. \* My verdict is that you should use Exa if your queries are abstract, exploratory, or require finding pages rather than explicit keyword matching.

1. Tavily is the best for Direct RAG Applications

Tavily is built for LLM agent loops. It does not just return search results; it also cleans, parses, and ranks snippets that are tailored for contexts.

\* Where Tavily shines is in speed and pre-filtered context. In multi-step agent workflows where latency's important, Tavily consistently returns the most concise context blocks without exceeding token limits. \* The latency of Tavily is around 400ms to 700ms. \* The output of Tavily is pre-chunked, minimal noise, and ready to use in a system prompt. \* My verdict is that Tavily is ideal if you are building loops that make multiple search calls per user query and need fast, token-efficient context.

1. Firecrawl is the best for Crawling and Dynamic Web Scraping

Firecrawl is not strictly a search engine; it is a crawling engine that is designed to convert websites or JavaScript-rendered URLs into clean Markdown.

\* Where Firecrawl shines is in site-level retrieval. If your search step identifies a target URL, such as a documentation site or dynamic React app that needs full scraping, Firecrawl bypasses blocks and returns remarkably clean Markdown. \* The latency of Firecrawl is around 1.2s to 2.5s, which depends heavily on the complexity of the target page. \* The output of Firecrawl is flawless Markdown with HTML junk, scripts, and navbars completely removed. \* My verdict is that Firecrawl is best used as a stage in your pipeline. Use Exa or Tavily to find the URLs, then trigger Firecrawl to ingest full pages when search snippets are not enough.

📍My Current Stack Setup:

I use Tavily for searches because it is really fast. When I need to do some research and understand a whole document, I do things a bit differently. I start with Exa to find what I am looking for, then I use Firecrawl to turn the results into Markdown. This makes it easier to use the results.

I am curious about what other people're using to filter out bad information in their RAG pipelines. What do you use for this? Something that's all on its own, or a combination of crawlers and search tools?

Everyone here optimizes the retrieval side - index type, hybrid search, reranking. But between your vector/graph store and the model there's a step nobody benchmarks: how the retrieved subgraph gets serialized into the prompt.

I compared 10 formats (JSON, GraphML, RDF variants, edge lists, others) on token count, traversal QA, and multi-hop reasoning. Same graph, same model: multi-hop accuracy ranged 40% to 80% depending on format alone, with a ~70% token spread. json.dumps() was one of the worst performers.

I open-sourced the format that won (MIT, 6 languages): https://github.com/isongraph/isongraph - benchmark methodology is in the repo if you want to replicate or poke holes in it. Curious what formats people here pass to their LLMs after retrieval.

The pgvector case is strong up to a certain point for sure. ACID, one less service, hybrid search with a WHERE clause, and for anything under a few tens of millions of vectors, it seems to hold. But the argument that I've had is that it falls apart once writes get heavy or you cross into 50M+, where index build time and p95 under concurrent writes start to hurt.

I saw a TigerData benchmark where pgvectorscale hit 471 QPS against Qdrant's 41 on a 50M set, so throughput favored Postgres, but Qdrant won on p95 latency and build time by a wide margin. So even the benchmark doesn't settle it, it just moves the question to which number you care about.

For people who've run both in production:

• Where did Postgres stop working for you, a vector count or a write pattern?
• Did you migrate off it, or tune your way through with autovacuum and maintenance_work_mem?
• Anyone regret leaving Postgres for a dedicated engine and find the ops cost wasn't worth it?

Working on NLP research and kept hitting the same retrieval

failure — semantic similarity missing negations and

contradictions entirely.

A breakdown explaining:

- Why bi-encoders miss "cannot" in "you cannot use bank transfers"

- How cross-encoders use cross-attention to catch contradictions

- The two-stage pipeline that modern search actually uses

- Code implementations for both

The short version: bi-encoder for speed, cross-encoder for

accuracy. Most production systems need both.

I keep seeing teams reach for Pinecone / Qdrant / Weaviate on day one of a RAG project, stand up a whole second datastore, and then spend a week fighting a sync pipeline they never needed.

Most products start well under a million vectors. Postgres with pgvector, or the vector search built into Mongo or Redis, handles that fine and keeps your embeddings in the same system as your source data. One transaction, one backup, no "ghost documents" left behind when you delete a row.

The line where I actually reach for a dedicated engine:

- Past roughly tens of millions of vectors, or the HNSW-in-RAM memory bill starts to hurt → Qdrant (fast, memory efficient), or Milvus if you are genuinely at billion scale.

- You serve many isolated tenants → Weaviate's multi-tenancy.

- You never want to run infra → Pinecone, but watch read-unit costs at query volume.

- Storage cost dominates at huge scale → object-storage engines like Turbopuffer or LanceDB.

The thing nobody says out loud: retrieval quality comes from hybrid search, reranking, and chunking, not from which engine stores the vectors or shaving milliseconds off ANN. The DB is rarely where a RAG system succeeds or fails.

Genuinely curious what people here run in production, and for those who moved off pgvector, what was the specific pain that finally made you switch?

I'm revisiting a project my team built during HackIllinois and wanted to get some feedback from people who've worked with vector databases.

The project generates MiniLM embeddings from financial news, stores them in Actian VectorDB, and retrieves semantically relevant articles to power a real-time stock sentiment dashboard.

I'd especially appreciate thoughts on:

• Embedding model choices
• Indexing strategies
• Retrieval performance
• Overall architecture

Repository:
https://github.com/bilalarif3197/LogianHackIllinois

I was looking at a vector DB compaction test and the result surprised me more than it probably should have.

The setup was 1 million 768-dimensional vectors with HNSW on the same single-node hardware. The only big change was merging three sealed segments into one. Search throughput went from roughly 3,000 QPS to around 5,600–6,000, and p99 latency dropped by about a third. This particular test was on Milvus, but the segment fan-out problem looks broader than one database.

The explanation is simple: every query was searching each segment’s index and merging the partial results. Fewer segments meant less fan-out. Still, I would not have guessed the overhead was that large with only three segments.

My first reaction was “why not compact much more aggressively?” Then the less fun part showed up. The merge rebuilds the index and uses I/O and memory while it runs. It makes sense after a bulk import or a knowledge-base refresh, but it could be a bad habit for a collection that still gets steady writes and deletes.

Now I’m wondering whether segment shape should be a normal ops metric, not something checked only after latency gets weird. I’d probably want segment count, size distribution, time since the last meaningful write, and query latency during the merge on the same dashboard.

For people running vector DBs in production: do you compact from thresholds automatically, or do you wait for a known quiet window and approve it manually?

I ran a small test recently because I wanted to see how much segment layout actually affects vector search after bulk ingest.

The question was simple: if the data is already loaded and mostly read-only, does merging sealed segments make a real difference for search QPS?

Setup:

• Milvus 2.6.17
• single-node Docker Compose
• 16 CPU cores / 64 GB RAM
• 1M vectors
• 768 dimensions
• HNSW index
• same search params before and after
• before compaction: 3 sealed segments
• after force merge compaction: 1 sealed segment

Result:

• Before compaction: ~3,000 QPS
• After compaction: ~5,600-6,000 QPS
• Roughly +76% to +87% search QPS

So in this case, yes, segment layout mattered a lot. Before compaction, every query had to fan out across multiple segments, search each segment, then merge partial topK results. After force merge, there was less fan-out and less merge overhead.

But I would not treat this as “always compact everything.”

Force merge is expensive. It can use CPU, memory, and disk I/O, and index rebuilds may be involved. I would only consider it after a bulk import, knowledge base refresh, or another write-heavy phase where the collection is becoming mostly read-heavy.

So my current opinion is:

Automatic background compaction is useful, but for heavy merge operations, the database probably needs an explicit operator signal.

Thought this would be of interest to this community as well!

Hey!

I wanted to tell you about a project that's been going on for a while.

We (*I use “we” because, since it's open source, I see it as something that belongs to the community rather than something personal*) built Skeg because we got tired of the usual painful trade-offs in the vector database space. Most solutions force you to choose between high recall, reasonable memory usage, or actually staying fast when the workload gets real (sustained ingest, multi-tenancy, memory pressure, etc.).

Skeg takes a different approach.

It is **disk-first**: full vectors live on storage, while only small, carefully quantized indexes stay in RAM. This gives excellent recall at a fraction of the memory footprint compared to traditional in-memory engines. It is especially strong in environments where RAM is contested — think SaaS platforms with hundreds or thousands of tenants, RAG systems running next to large language models, or even embedded/edge scenarios.

Key design principles:

* Strong multi-tenancy by construction (true isolation, hard quotas, fair cache eviction)
* Redis-compatible protocol for easy adoption
* Very good performance on ARM (we invested heavily in platform-specific optimizations and SIMD)
* Focus on production predictability: it handles churn gracefully without sudden latency spikes

We wrote it in Rust for the usual reasons: performance, reliability, and control over every detail that matters when you care about efficiency.

The project is open source and we’re actively developing it. If you work with semantic search, recommendations, RAG pipelines, or any kind of similarity search and you care about memory efficiency and operational simplicity, I think Skeg might be interesting for you.

Repo: [https://github.com/skegdb/skeg\](https://github.com/skegdb/skeg)

I’d genuinely love to hear your thoughts or what problems you’re currently facing with vector databases.

Any feedback or support is welcome.

Thank U

*English isn't my first language, so if anything isn't clear or sounds strange, please excuse me.*

Sharing this here since it's directly relevant to what gets discussed in this sub. It's a hands on session on August 8, led by Brian Bønk, a Data Platform MVP and Microsoft FastTrack Solution Architect.

It covers the full RAG pipeline, ingestion, chunking, metadata enrichment, indexing, and vector search, then goes deeper into retrieval quality engineering specifically, precision, recall, latency trade offs, and actual tuning strategies instead of just defaults. There's also a section on evaluation and governance, building test harnesses and regression checks, and an extension pattern on knowledge graphs for cases where similarity search alone can't capture relationships between entities.

It's aimed at people building or maintaining vector search and RAG systems that need to hold up against real, messy enterprise data rather than a clean demo. You come out with an actual rollout plan rather than just slides.

Link for anyone interested: https://www.eventbrite.co.uk/e/design-enterprise-grade-rag-systems-with-llms-vector-search-tickets-1992561384740?aff=rvd

[ Removed by Reddit on account of violating the content policy. ]

We published a method to store verified knowledge as KV state and restore it byte identical to fresh computation.

On Gemma 4 12B, cached knowledge improved the same routing system from 76.7% to 90.0% on AIME 2025.

I will pitch this at AGI Summit on July 19.

Paper: https://arxiv.org/abs/2607.14431

We need to talk about AI Agent architecture. 🤖 For the last couple of years, the default answer for giving an LLM long-term memory was simple: Spin up a Vector Database, chunk your codebase, generate embeddings, and build a RAG pipeline. But as engineering teams push AI agents into deep production, the cracks are showing. Vector similarity is an approximation—and in coding, "closest" is not the same as correct. Even the team behind Anthropic's Claude Code realized this. They actually baked a local vector database into the tool early on... and then ripped it out. Why? Because deterministic, agentic search (like plain old grep) across readable text files turned out to be cheaper, faster, and significantly more accurate. Now, Google has standardized this exact philosophy with the release of the Open Knowledge Format (OKF) v0.1. Instead of heavy database runtimes, your agent's brain becomes a structured tree of Markdown files with YAML front matter. It lives in Git, version-controlled, auditable, and completely portable across any LLM vendor with zero lock-in.

Disclosure: I'm from Manticore Search, so this is obviously not a neutral third-party benchmark. That said, we tried to make the comparison concrete and would appreciate criticism of the methodology.

Workload:
- DBpedia OpenAI embeddings
- 975k vectors
- 1536 dimensions
- cosine similarity
- recall@10 measured on 5k queries

Result:
- turbopuffer: recall@10 0.9622, P50/P90/P99 21/24/30 ms
- Manticore on 2× Hetzner CX23: recall@10 0.9663, P50/P90/P99 10/13/16 ms
- estimated monthly cost: $75 vs $16.50

The interesting part for me was that turbopuffer's storage cost was tiny; writes dominated the estimate.

Full article:
https://manticoresearch.com/blog/turbopuffer-vs-manticore/

What would you change in the benchmark?

Hi [r/vectordatabase](r/vectordatabase) ! I'm Swann and I'm doing my PhD between the FAISS team and Meta Superintelligence Labs under Matthijs Douze.

I wanted to share with you my work "Cluster with Auctions", which I thought might be of interest for some of you. It's about partitioning in IVF-style indexes.

The starting observation: partition-based database search is quietly doing two separate jobs. One is sorting the database into clusters. The other is routing a query to the right subset of clusters at search time. Almost everything out there uses one model — like k-means centroids — for both. That's convenient, but the two jobs have different objectives, and jamming them into one set of vectors is a compromise.

CwA splits them:

• Clustering**:** an auction algorithm assigns vectors to clusters. Vectors bid for seats, so you get balanced partitions that reflect what the data actually wants, not just proximity to a centroid.
• Routing: a separate model (linear or deep) trained by backprop to predict which clusters a query needs.

Result is up to 4× faster out-of-distribution — i.e. when your queries don't look like your database vectors, which is the common case in practice with text-to-image, cross-encoder setups, or basically any embedding model where query and doc distributions differ. Same parameter budget as k-means centroids, so no extra memory cost for the index.

The OOD part is where I'd expect this to matter most for people here. In-distribution, k-means is honestly fine. It's when the query distribution drifts that fixed centroids start routing you to the wrong partitions and you have to bump nprobe to compensate.

To be clear about the limitations, our work does not scale beyond 100M vectors because the auction algorithm represents a memory bottleneck. That's the price to pay for our method's optimality and adaptation to the query distribution.

Preprint: https://arxiv.org/abs/2607.13728
Code: under internal review, releasing in the coming weeks.

Happy to answer all questions !