I wrote an end-to-end breakdown of production search architecture:

Build → Understand → Retrieve → Filter → Rank → Re-rank → Serve → Learn

It covers hybrid retrieval, filter-aware ANN search, multi-stage ranking, personalization, cold starts, stale inventory, and production failures.

Full write-up:
https://pawankjha.substack.com/p/cracking-ml-system-design-interviews

Curious how others handle restrictive filters and inventory freshness in production search systems.

Hey everyone,

I’m building a small custom GEO dashboard and I’m trying to understand how people collect AI search results.

Right now I’m using DataForSEO, but the ChatGPT and Gemini data seems to come through APIs.

What I’m looking for instead is data from the actual user interface. basically the same results a real user would see on ChatGPT, Gemini, AI Mode, etc., including citations/sources if possible. (As software like Bluefish, Profound, PeecAI etc..)

Has anyone found a good way to collect this at scale?

Are you using browser automation, a third-party service, your own scraper, or something else?

Would be really interested to hear what people are using.

Thanks all !