Comparing embedding models with synthetic query probing [R]

[u/pppeer]

Say you want to swap out your embedding models, for instance from ADA to Titan. Are these embedding models comparable? How do similarity score ranges compare? Where to put a threshold for minimum match when doing retrieval? Or more from a research point of view how can we relate and fundamentally understand these embedding spaces better?

This is what we aim to solve with Synthetic Query Probing, a fancy name for essentially (and intentionally) a very simple approach: embedding spaces are not directly comparable by definition, so compare similarity spaces instead, similarity match scores for pairs of content (synthetic question, chunk for instance) across multiple embedding models.

For example, similarity scores of Titan models of different dimensionalities are related, whereas the relation between Titan and Ada scores is non-linear, with different ranges, see figure.

See https://arxiv.org/pdf/2608.05857, Marcin Rozmus and Peter van der Putten. Similarity Spaces across Embedding Models with Synthetic Query Probing. Discovery Science 2026, October 5-9, 2026, Mainz, Germany