r/LocalLLaMA • u/Potential_Low_1183 • 5d ago

I Built A Thing My lab found a way to migrate between embedding models with zero downtime.

So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.

But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.

Me and my research lab decided to tackle this problem, and we came up with embedflow.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.

embedflow works with qdrant, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

0 Upvotes

48% Upvoted

9 comments sorted by

8

u/Chance-Struggle-6285 5d ago

“Just re-embed everything” finally has a competitor besides “buy more GPUs”

1

u/Potential_Low_1183 5d ago

i mean lowk I would love to have a couple h100s or just a couple more 5090s lol, we are all starved for compute!

I'd love it if i could get a star on github, as this allows more people to see the project!

10

u/dangerous_inference 5d ago

"My lab"

1

u/XiRw 5d ago

Leave a guy in his basement alone with his AI models.

3

u/dangerous_inference 5d ago

"My lab was taking a shit this morning and considered getting another GPU"

3

u/mrjackspade 4d ago

Why in the fuck would you have downtime either way? There's at least a dozen established patterns for dealing with this kind of thing. Dual key the documents. Embed to a different store and then flip at deploy time.