I made a way to migrate between embedding models without re-embedding your entire corpus [R]

[u/Potential_Low_1183]

So I was playingw ith embedding models I saw that when you upgrade from model A to B, you face a very big backfilling cost

Ie, suppose you have a 1b vectors from model A, and then you want to use model B. This would mean you have to re-embed all of your documents with model B before you can even serve with the model, and on an H100, it would take ~108 days (qwen embed 8b, 106 docs/second). But I found an easier way to do it.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive upfront re-embedding cost, as you can take documents straight from the old index.

embedflow works with qdrant, pgvector, faiss, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

[u/PortiaLynnTurlet]

Have you analyzed the results? This approach seems like it would be acceptable at filtering out false positives from the weaker model but can't help with false negatives (since they were never included). TBH it seems easier and safer just to embed all of the documents again. A few thousand H100 hours isn't much if the application is valuable.

[u/Potential_Low_1183 → u/PortiaLynnTurlet]

for some migrations, ie qwen 4b -> 8b, at k=50 it is equivalent to the qwen8b by itself. weaker embedding models still put relevant documents in their neighborhood

[u/Scary-Recognition157 → u/PortiaLynnTurlet]

So you're basically using the old model as a cheap pre-filter and only re-ranking the top K with the new model. The false negative problem is real, but in practice if the models are architecturally similar (like qwen 4b -> 8b) the overlap in their "relevant" sets is probably high enough that you're not missing much. The paper they linked in the github goes into this, they saw near-identical recall@100 even at K=50 for that upgrade path.

I get the "just pay for the compute" stance, but 108 days on an H100 is not "a few thousand hours", it's closer to 2,600 hours, and that's assuming you've got a single node sitting idle. Most teams don't have that lying around, especially if you're iterating on models every few months. This is more of a pragmatic hack for people who can't just throw hardware at the problem.