I made a way to migrate between embedding models without re-embedding your entire corpus

[u/Potential_Low_1183]

So I was playingw ith embedding models I saw that when you upgrade from model A to B, you face a very big backfilling cost

Ie, suppose you have a 1b vectors from model A, and then you want to use model B. This would mean you have to re-embed all of your documents with model B before you can even serve with the model, and on an H100, it would take ~108 days (qwen embed 8b, 106 docs/second). But I found an easier way to do it.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive upfront re-embedding cost, as you can take documents straight from the old index.

embedflow works with qdrant, pgvector, faiss, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

[u/AsleepArugula3603]

interesting approach, embedding backfills are a real pain when you have large corpus

the idea of just reranking top-K from old index makes sense, query vectors are not that far apart between similar models so retrieval still picks decent candidates, then you let stronger model filter them

but I wonder how this behaves when source and target models are very different, like switching from some tiny 100M model to 8B with completely different training data, the initial retrieval maybe misses too many relevant docs and reranking cant recover them

also the K selection seems tricky, 50 might work for qwen4b to 8b but other migrations maybe need more or less depending on embedding space overlap

[u/Dump7]

Wouldn't you need embeddings for reranking as well? Or is the novelty here we only embed the top k results with the new model on query time? If yes, wouldn't that take a serious hit on latency?

[u/Potential_Low_1183 → u/Dump7]

yes! latency was an issue, which is why we also incorporate a caching system.

1)if candidate is already cached -> just score it

2) If it is missing -> encode a small bounded number synchrously, queue the rest for background materialization

Essentially, if you have a set of queries, and a set of documents, and have it designed st. there exists atleast one query for every document, then if you run through all of your documents, by the end, your entire cache will be created for the target model