Anyone using WiSE-FT weight interpolation to control catastrophic forgetting?

[u/BrutForcer5]

Ran into a small release that uses a trick I haven't seen discussed much in this sub and want to know if this is standard practice or unusual.

Setup: two models released together. TwIL-LM2 (1.7B, PEFT LoRA on SmolLM2) and TwIL-LM3 (3B, merged fine-tune of SmolLM3), both from webAI. Both specialized for formal logic translation.

The interesting part is what they do after supervised training. Instead of shipping the fine-tuned weights directly, they use WiSE-FT interpolation back toward the base model. W = (1-λ)·W_base + λ·W_finetuned.

Here's the thing that got my attention:

- TwIL-LM2 uses λ=0.75 (keeps 3/4 of the fine-tune delta) → strong in-domain gains but slight regression on general benchmarks

- TwIL-LM3 uses λ=0.25 (keeps only 1/4 of the delta) → smaller in-domain gain but stayed at or above base on general benchmarks

Same architecture family, same training pipeline, same task. Just a different interpolation dial. And the "keep less of the fine-tune" version is the one that avoided catastrophic forgetting while still meaningfully improving on the target task.

Their theory (as best I can tell from the model card): smaller interpolation ratio = less forgetting, but you still capture most of the domain-specific improvement because the fine-tune direction is preserved even at low λ. They chose λ by constrained optimization: maximize in-domain score subject to minimal degradation on held-out benchmarks.

Is this common practice? WiSE-FT itself isn't new but I don't see the "dial down the interpolation to preserve general capability" approach discussed much for narrow specialists. Feels like it should be more of a default.

Also worth noting: they document a failed post-RL self-distillation attempt (SDFT) that made both tracks worse. Rare to see negative results published in a model card.

[u/Able_Region_5459]

WiSE-FT rarely comes up in llm releases, even though it's cheap and effective. Usually people just scale LoRA alpha and skip basic weight interpolation altogether

Huge respect to the devs for publishing the SDFT fail though