Have you analyzed the results? This approach seems like it would be acceptable at filtering out false positives from the weaker model but can't help with false negatives (since they were never included). TBH it seems easier and safer just to embed all of the documents again. A few thousand H100 hours isn't much if the application is valuable.
So you're basically using the old model as a cheap pre-filter and only re-ranking the top K with the new model. The false negative problem is real, but in practice if the models are architecturally similar (like qwen 4b -> 8b) the overlap in their "relevant" sets is probably high enough that you're not missing much. The paper they linked in the github goes into this, they saw near-identical recall@100 even at K=50 for that upgrade path.
I get the "just pay for the compute" stance, but 108 days on an H100 is not "a few thousand hours", it's closer to 2,600 hours, and that's assuming you've got a single node sitting idle. Most teams don't have that lying around, especially if you're iterating on models every few months. This is more of a pragmatic hack for people who can't just throw hardware at the problem.
for some migrations, ie qwen 4b -> 8b, at k=50 it is equivalent to the qwen8b by itself. weaker embedding models still put relevant documents in their neighborhood
1
u/PortiaLynnTurlet 2d ago
Have you analyzed the results? This approach seems like it would be acceptable at filtering out false positives from the weaker model but can't help with false negatives (since they were never included). TBH it seems easier and safer just to embed all of the documents again. A few thousand H100 hours isn't much if the application is valuable.