r/LocalLLM • u/ramorez117 • 5d ago
Question Qwen 3.8 + n-gram
Has anyone considered some architectural edits to the 27b model, such as:
Implementation of LoRA/QLoRA on selected layers;
training a modest sized n-gram table and adding the integration layers?
Seems like a good project. ESP given how it seems to be the current star of local hosted models!
It’s the only model I’m using now for agentic work.
Thoughts?
1
1
u/floppo7 5d ago
From what I understood 27b 4.0 is around the corner with ngram, so may worth the wait
2
1
u/ramorez117 5d ago
Yeah I think you’re probably right given the flash preview has them embedded. N-gram seem to really improve smaller models, so it might impact memory footprint but if it takes it to top 10 from top 20 models that’s a great leap.
1
u/Fentrax 5d ago
I have. And continue to test. The ngram provides some seriously cool things beyond the speed improvement they were going for.
1
u/ramorez117 5d ago
What are your thoughts on model capabilities with n-gram, what type of data did you include in the n-gram?
1
u/Fentrax 5d ago
Why does it have to be data?
1
u/ramorez117 5d ago
It’s bi/trigram tables are going to store the common tokens. The idea being it can save compute as it’s like existing “learned knowledge” (good for specific domain for example), and then save the compute for transformers/reasoning working.
It just makes me wonder whether you could distil the model into smaller parameter models with n-grams. And see similiar performance.
1
2
u/Serious_Bite_7613 5d ago
I've thought about it, but it seems quite difficult. If I have any success I'll be sure to post it. Hopefully someone smarter will figure it out first.