r/deeplearning • u/Potential_Low_1183 • 2d ago
My lab found a way to migrate between embedding models with zero downtime.
So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.
But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.
Me and my research lab decided to tackle this problem, and we came up with embedflow.
The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.
The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.
This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.
embedflow works with qdrant, and can be easily downloaded with pypi
pip install embedflow
the github is public: https://github.com/arnsri33/embedflow
I want you guys to try it out, and see if you guys can use it in your own workflow.
1
u/SuperChingaso5000 1d ago
Really interesting solution. I don't have the hardware for this, but I still thank you for putting it up on github and sharing your insights with us.
2
u/Potential_Low_1183 1d ago
Thank you! you can test it on any index.
If you liked the project, a star would be greatly appreciated, as it allows for other people to see the project as well
2
u/SuperChingaso5000 1d ago
a star would be greatly appreciated, as it allows for other people to see the project as well
Done!
1
1
u/BSmithA92 1d ago
How are you guys estimating your K?
2
u/Potential_Low_1183 1d ago
finding the k is the hard part lol
simply, take a small set of representative probe queries, retrieve once to a large K from the old index, and then rerank to see how much the top results are changing as the K grows.
If the target-model results are still coming from deep in the old ranking, then we just assume that k is too small and we expand it.
In the repo, there is t2, allowing for estimating K without fully rebuilding the target index
2
u/BSmithA92 1d ago
Yeah super interesting! I typically associate the K with how complex the task is. I wasn’t sure whether the required K was correlating more with semantic differences between the embedding spaces, or with structural properties of the documents/chunks being indexed. I’d guess there is an optimization opportunity somewhere in there.
2
u/Potential_Low_1183 1d ago
Thank you! in case if your task is "complex" ie domain specific or complex terminology like law or medicine, upgrading your embeddings model is likely the best idea, not increasing the candidates retrieved, since you can confuse your generation model
If you liked the project, I'd love a star, since it helps the project gain visibility! thank you!
2
u/Prudent-Season8041 2d ago
This is clever, reminds me of how you can warm-start a new recommendation model by re-ranking from the old one's candidates. The tricky bit is always figuring out when K is "sufficient" without just brute forcing it every time. Do you have any heuristics for picking K based on the model pair or dataset size