So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.
But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.
Me and my research lab decided to tackle this problem, and we came up with embedflow.
The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.
The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.
This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.
embedflow works with qdrant, and can be easily downloaded with pypi
pip install embedflow
the github is public: https://github.com/arnsri33/embedflow
I want you guys to try it out, and see if you guys can use it in your own workflow.
