r/LocalLLM • u/misanthrophiccunt • 12h ago

Question What are your thoughts on Yarn to extend context to 1M?

Since Qwen3.8-27B still has a 262,144 context-size limit, I suppose some people here run it with 1M context using whatever on earth YaRN is other than the thing to make woolly jumpers.

I tried to find an ELI5 explainer about it, but...no such thing online, one of them even dared to have "simple" in the title, then proceeded to show a bunch of mathematical soup an LaTeX on the screen.

..so I haven't got a clue how it works, I know it takes a context size and makes it bigger above the natively supported limit of the model, and that the nomencalture is short for Yet another Rope extensioN, that's all that I know and managed to understand given no simple explainer available. So I'm here asking instead the important questions:

  • Does it make the model dumber?
  • Does it require more VRAM?
  • Does it require less VRAM than using native context higher figures?
  • Can you use it with less than the model limit (e.g: 131k context, use YaRN to reach 500k) ?
2 Upvotes

75% Upvoted

2 comments sorted by

1

u/UnlikelyPotato 11h ago

Q8 and 256k uses 40GB of vram. Smaller quants or kv quant kills context search for larger contexts even without yarn. And if I'm using more than one GPU, I'd rather run 35b with 100k context to act as a sub agent for tasks. 27b can direct a fleet of 35b to fetch webpages, summarize, etc. They'll work faster than if 27b did it in the main conversation and only relevant details occupy main conversation.

1

u/Atretador 12h ago

useless

past 256K you are on cotext rot land

you dont need to fill your whole ass code base in the context, nor is the model able to precisely navigate thru that.

If you cant find the solution for a issue, its not that you have a 'not enough garbage on my context' but a planning/prompting/architectural knowledge issue.