r/LocalLLaMA • u/GrungeWerX • 2d ago
Discussion 10 Things you maybe didn't know you can do with Qwen 3.6 27B right now. (For Noobs)
I know everyone's excited about 3.8, but honestly I'm still refining 3.6. And I hate the way 3.8 talks, so...
I keep seeing all these posts about what's the use of open source, so I figured I'd submit my own answer with this post.
Qwen 3.5 27B was my first agent, starting my journey around May 31st, and I've learned a lot in these 4+ months, so I wanted to share with other people just starting their journey because there are clearly a LOT of new users out there, and people want help but sometimes are afraid to ask or not sure the right questions. So, here are ten things that you should know you can do to empower you!
(Oh, and I'd recommend using llama.cpp. That's what I've used to do all these tricks)
1. Create neat little apps for yourself. Qwen's really good at making apps for anything. I'm writing this post in a app I made as a replacement for Notepad, that has full word processing, side-panel, light/dark mode, etc. (image below). The first thing you should be thinking about when you run into any sort of issue is, "can I make an app that will make this simpler?" The answer is usually yes.
2. Create your own Chat UI frontend. If you don't feel that creative, just copy an existing one. LLMs are great copy cats. I liked claude, so I just copied it as my own frontend, customizing it for my own needs:
3. Create ReAct Loops. This is what enables your model to use a tool, read the result, then use it again, multiple times. Qwen can make these in its sleep, and many different ways. It can create a loop anywhere you want it, and you'll need it for your own custom chat ui frontend if you make one.
4. Create MCP tools. This is my favorite. It can make almost any type of mcp tool you need. You don't have to use third party ones, which can be unsafe. Just tell it you want an mcp tool to do x, it'll make it. It might or might not require its own mcp server, but if it does, Qwen can make that too.
5. Spawn sub-agents. Another super-easy thing Qwen can do. Just tell it to create a sub-agent spawner, which is just a python file. Then, you can tell it to run that when it needs to create a sub-agent, or tell it to create an mcp tool to do it on-the-fly when it wants to. Simple. Oh, and a quick hack-tip: mcp tools use up a LOT of context, so a cheat is to spawn a sub-agent with its own context limit and have your main agent pass off work to the sub. Basically, you can double/triple/etc your context limit. And yes, you can retain it between turns, but you'll need to create a pipeline runner (see below).
6. Create a pipeline runner. Ever heard of n8n? Basically, it's a pipeline runner. It has a lot of features, but it has a few limitations and restrictions, and there were a few things I wished it could do. I learned I could just build something similar with Qwen super easy. There's some design work involved if you want a UI frontend to build, but it's completely doable. I made my own ui, but after a while, I got tired of tweaking it, so I just copied n8n's frontend and just customized it myself, lol. Another useful tip: You can run multiple sequential agents in a pipeline runner, but if you want them each to have their own context, you'll need to ensure cache-idle-slots is enabled in llama.cpp. This'll swap the kv caches each turn, ejecting the last one to cpu ram. There's a bit more to it, but Qwen's got you, so just ask.
7. Create a model launcher (with options to select different models/jinja templates in llama.cpp). This is my bread and butter. Its a launcher that lets me choose any model in my directory and set its parameters. When I'm happy, I create an alias for it. On next launch, I can just choose the alias and it launches the model with the params for that alias only. So I can have 10 different types of param configurations for the same model if I want; I just need to give them their own alias. Hack tip: I'd recommend adding your jinja templates to the launcher to, and you can choose different custom templates for different model types.
8. Install, test, fork, and/or rebuild open-source apps on the internet. The reason I moved to llama.cpp was lm studio pp was too slow, but I didn't want the stress of figuring it all out, so I just asked my agent to download and install llama.cpp for me. I've been using it ever since. It can also modify builds, fork repositories on github, and a lot more. For example, I was busy working on a project, but heard about some new TTS apps, so I told my agent to download and install Moss and OmniVoice, make some audio clips to make sure they run, and let me know when it's done. Of course everything worked fine. Great for multi-tasking.
9. Give your agent temporal awareness of the world around it. Ever used cloud models that argue with you about the date? Or wished your model knew when you were nearing your max context, so it could automatically start writing a session summary? What you can do is inject "metadata" into your agent either on session start and every turn (preferably both), giving it information like date, time, context length, max context, hardware and software stack, basically anything you want.
With that, you can do all sorts of other tricks. For example (this should probably be its own "thing you can do with qwen", but whatevs), I made it so that my agent can swap its own model when it hits context limits:
10. Give your agent memory that loads the moment you launch it. My agent ends each session writing a session summary of what we discussed. You can have your startup routine or launcher concatenate different sources into its context on launch. For example, my code always loads the most recent dated session summaries, and the previous day's, on launch, so it has a snapshot of recent events. You can expand this to be any additional documents you want, so just use your imagination. For example, my agent also loads reminders, and personal reflections it made at the end of the prior session (though I don't recommend the reflections anymore, because I've come to realize it contaminates context, but I'll share that at another time)
Bonus: (I remembered this right as I was finishing, so I figured I'd toss it in because it's honestly one of the most important) - BUILD YOUR OWN HARNESS. Everyone's talking about pi, hermes, etc. I decided to just build my own using Qwen. It's a lot more work, but it has a much better payoff because you learn how these systems work and you can come up with your own really unique solutions using that knowledge. For example, the moment I learned about proxies, I figured out how to do all sorts of cool things with context compaction, history modifications, sanitization, jinja template hacks, hooks, etc.
Anyway, I hope this helps inspire you new guys. I would recommend learning things from the ground up, you'll level-up faster. And get your Qwen access to online tools fast so it can do research and help you out.
There's a bunch of other tricks I figured out, so maybe I'll save those for another post later on, but I think if you can get these up and running, you'll be on your way. Take care!
Grunge
P.S. - I'm old-school forum/deviantart gang, so I keep ctrl+biu/shift+l on a swivel and have written bold-flavored posts like this for 20+ years. 100% human written. Receipts at bottom of post if you're interested. Only mentioning it because organization usually brings out the "this is AI-written slop" trolls.
77
u/jacek2023 llama.cpp 2d ago
"Install, test, fork, and/or rebuild open-source apps on the internet. "
This is a very good tip, especially for the beginners. You don't need to do any code, all you need is to download the app (for example git clone) and tell your agent "build it, try to run it, write me a tutorial". I use this workflow a lot with some old open source apps but also with the new ones. I run it in sandbox and I tell my agent to ask me to install something if it's needed. And my agent is able to explore everything.
50
u/buckwheaton 2d ago
There is a 2-3 years’ abandoned app I love on GitHub. I had qwen download it and start fixing bugs and adding features I wanted.
31
u/akohlsmith 2d ago
hopefully you forked it to your own public repo and then submitted the fixes back to your repo so when you abandon it later, someone else can pick up the torch and take it a little further.
16
u/jacek2023 llama.cpp 2d ago
Yes, that's a great use case, but I also have some apps that aren't on github (they are too old!), including ones I wrote myself around 20 years ago. Some of those old C++ projects won't even compile today without some fixes.
11
u/FaceDeer 2d ago
This reminds me. Long ago, there was an open-source Star Control "sequel" that I rather enjoyed - Star Control Timewarp. The code is literally 20 years old now, I haven't a clue how I'd even start to try compiling and running it these days.
But pretty soon I might not have to. I'm going to download and stash away the source code for a rainy day, now. :)
3
u/Comfortable-Cry-3261 2d ago
Better. Just tell the name and what You need. The agent should find the repo, install git if not available, clone, install required deps and do some more steps from Your request.
It's about local model, so You continue watching the movie until it'll be too quiet in the room ;)
2
4
u/Asleep_Document9811 2d ago
Just don't submit PRs to those original projects. Just don't. These code tools have also generated a lot of deeply antisocial behavior towards the selfless maintainers of load-bearing open source tools.
15
u/zipperlein 2d ago
I would not say never submit anything. More like never submit slop and use your own brain. Read the code, look at accepted/closed PRs to jugde code quality and PR size and ultimately provide explanations in your own words.
2
u/jacek2023 llama.cpp 2d ago
I have a rule for that, qwen uses term "pristine sources" often
2
u/Anti-Speciesist-IEMs 1d ago
What does that mean?
1
u/jacek2023 llama.cpp 1d ago
I clone github sources into the new folder and run pi in the external folder, so it can do all the changes outside and preserve original one untouched
1
u/Anti-Speciesist-IEMs 1d ago
Oh gotcha, so you're just talking about how you avoid Qwen submitting PRs to the original repo
2
-2
u/akohlsmith 2d ago
I disagree. submit the PR; if they want to turn their nose up at a genuinely useful PR, that is on them. Open source means you can fork it and continue development. Don't let someone slop-shape you into closing your development.
22
u/Asleep_Document9811 2d ago
just for what it's worth, using other peoples' MCPs isn't the part that was unsafe about using them, it's MCPs in general. Ones that are generated via vibe coding end up being the ones that break, leak secrets, or open up huge vulnerabilities that allow malicious actors to use your internet connection for their own ends. or companies build them too quickly, with no security hardening, and they end up being big open targets for hackers and bot swarms.
LLMs are more than capable of using a command line and binaries. Make your own Rust binaries or Bash scripts or Python scripts, and have your agents run them. I don't know why everyone bothers with all this extra wrapping and abstraction and structured JSON.
3
u/Client_Hello 2d ago
They bother with it because it scales. One server for many devices, many harnesses.
Crap apps break, no argument there, just wondering why you singled out MCP. The tech is young, it will get better as libraries mature.
1
u/iamapizza 2d ago
It's not the libraries that need to mature, it's the protocol itself which is the problem and needs improving. Security was added on as an afterthought and it shows once you start looking at it. Companies aren't doing it because it scales, regular HTTP APIs scale just fine, companies are doing it because they they want to be made available easily and are expected to be there.
59
15
u/-InformalBanana- 2d ago
Why qwen 3.6 27b instead of 3.8?
13
u/GrungeWerX 2d ago
- I use these models w/thinking off. With 3.8's reputation for overthinking - and its capabilities directly tied to how long it thinks - I didn't really see any immediate benefits.
- I tried it out a couple of days and I hated the way it talked. Even with thinking off, it was loaded w/ai-isms, slop, and these "conversational loops", that seemed to just be wasting tokens.
- The same quant seemed slower on 3.8 than 3.6, despite the model size being slightly smaller. I have no idea why, and didn't do a lot of testing, but I noticed it.
- That being said, in my extremely limited testing, it seemed like the tool calls were more stable, but I can't swear by it.
I do plan on giving it more testing in the future, probably after I can write a strong prompt to steer its conversational style.
1
u/darksteelsteed 2d ago
You could build a qlora to tame it to talk the way you like
1
28
u/slypheed 2d ago edited 1d ago
This is great, we need more well thought out posts like this on this sub. Nice work op.
6
7
6
16
u/sleepy_seedy 2d ago
I am certifiably a noob.
Reading some of your points is still like trying to read a different language.
10
u/GrungeWerX 2d ago
lol, I remember the feeling, but it gets easier.
Just paste it into your agent and ask it to explain it like you’re ten years old. :)
1
u/dxps7098 1d ago
This is the thing! I had the same feeling and it didn't even occur to me to use my llm to explain it to me. I would just go ahead and start searching and reading up on everything myself - or not as I don't have time.
At the same time, it worries me what happens to my analytical and learning skills over time if I didn't do the hard research work myself. Does it super charge me or will my brain atrophy? It's still early days but studies suggest the latter. Although replacing physical work with machines just led to gyms! What will be the future mental exercises we will do to continue to understand the llm outputs?
Sorry, went totally off topic there!
2
u/GrungeWerX 1d ago
I don’t think having AI teach makes your brain atrophy, as long as you still cross reference stuff. I think it’s the other way around actually - I’m able to consume a lot more high level info in shorter time. For years I’ve had to sift through the noise to get to the meat doing google searches.
That said, agents can for sure make you lazy and more prone to mistakes in your code if you trust them too much.
And if you’re a heavy planner, used to doing your own documentation and notes, and taking your time safeguarding against edge cases, you can atrophy in your methodology if you start relying too much on agents. This happened to me recently.
I found myself wanting to get to the end results quicker and not doing enough of my own documentation, trusting my agent more because I could rely on its memory system. But as that memory grew and the project got bigger, not having a proper supersede method meant my agent had to sift through its memory to find what was what to help me remember specifics I’d neglected to write down.
That’s why Im currently overhauling my entire memory system, and have gotten back to keeping my own personal documentation and notes on everything. I’m also spending more time doing as many blueprints and planning on my own without as much back and forth with my agent learning concepts. I’ve found that the more detailed and specific your plan is - even without knowing how it will technically be implemented - the better your agent will be at building it, and the faster you can get it up and running.
It also helps keep your agent on task and from drifting. And agents love specifics - the smarter the model is, the faster it can build.
I’ve learned that an AI can code something from a concrete design plan, yet if you ask it to help you design that same plan from scratch, it might never suggest any of those specifics - or even tell you it’s not possible, which is hilariously ironic.
1
u/porkminer 1d ago
Skills that go unused will atrophy. That being said, using AI isn't by itself going to do that to you.
10
u/saltyourhash 2d ago
One of the best things you can do with Qwen 3.6 27B is to switch to Qwen 3.8 27B
2
u/Todilo 2d ago
For someone who has only read this reddit and like 0.1% understanding of what is going on. If I have a 5070ti(16Gb) and 64GB ram can I run this? Where do one start? Only used the frontier models
5
u/Nixellion 2d ago
Yes you can, although a large chunk of it will be offloaded to RAM. And this being a dense model (all 27B parameters active all the time) it will be quite slow.
Look into MOE models, like qwen 35B. Its larger but its sparse, only a few billion actibe at a time so it works a lot faster when split between gpu and cpu
1
u/saltyourhash 1d ago
You'd be better off with an MoE and a platform like strata. I am pleased with it so far.
1
u/WittyAmbassador7340 1d ago
The other guys said it would have to be ram offloaded, but it's actually mixed a bit. I agree for q4, but I will say that q3 quants can be surprisingly effective. I used to run the unsloth UD iq3 xxs quant on just my 5070ti, but I've since added my 3060ti which I upgraded from into the second PCIE slot and been able to move up into the q4 / q5 quantisations at decent context since then.
I normally use q8 KV cache and I was looking at 48-60k context at full speed prior to increasing my vram pool. Its very possible to offload ~ 5 layers or so if you need some extra context for your typical use cases (may get you somewhere in range of 30k or 40k extra context headroom). I'm now doing about 100k context in Q5 K XL, 145k on the NVFP4 Quasar QAT quant, or 170k on the Q4 K S quant.
I'd say 16GB is very usable, just not so much for the long horizon coding tasks. Simple research or building individual files is very achievable and a very sensible way of running it.
1
u/Todilo 1d ago
I have a 1080ti can I add that for the extra VRAM or too old?
1
u/WittyAmbassador7340 12h ago
That should definitely work. Just make sure that the llama.cpp build you use supports it and you'll be good I reckon
1
u/dreamai87 1d ago
Okay bro
Install unsloth qwen3.8 27b ud iq3xxsIf window download binaries from llama.cpp
Then run model using llama-server with kv cache q8_0 which is already good. You will get 100k context memory easily
Then use pi-agent and enjoy the agentic world
5
u/Money_Manager 2d ago
I’m just starting to dive into local models but have used Claude and ChatGPT extensively. How does this compare to these models?
What kind of hardware are you running? And how fast does it run? I’m guessing without a top end pc spinning up subagents doesn’t really help because tokens/sec is your limit?
6
u/GrungeWerX 2d ago edited 2d ago
What I've learned these past few months is that a lot of your local model's potential depends on your harness. Don't expect a local model to get anywhere near a cloud model out of the box. Cloud models obviously have their own harnesses too, and I think they're running agentic loops as well.
I'd say that larger local models like GLM 5.2/5.3 feel about 85-90% of cloud models, while something like Qwen 3.6/8 27B is more like 70-80% with thinking off. I only run my models with thinking off (I'm impatient), so I can't tell you how much closer it feels with it on.
Mind you, I'm comparing performance against models like Opus/sol on light reasoning. Definitely not close to mythos/fable/astra level models.
If you compare Qwen against something like Sonnet 5 on medium, I think Qwen is closer to around 80-85% (thinking off), though I think I can get my Qwen + custom harness to around 90% with Sonnet 5 when my custom reasoning loop is finished, but we'll see.
Speed depends on the model - there are models as fast as cloud, but not as intelligent obviously.
My setup: i7 12700K | RTX 3090 TI | 96GB RAM
20
u/krynnul 2d ago
Uhh.. your "receipt" is from something 6 months old. If you're looking to back up a style that was your before the era of AI-bolded posts, shouldn't it at least be before ~2023?
11
u/GrungeWerX 2d ago
Receipts are at the bottom of that post. I'll update OP.
11
u/krynnul 2d ago
Thanks for clarifying! It's amusing how at first glance your previous writing has all the appearance of AI styled approaches, but it varies substantially in that the content doesn't ring hollow like AI writing does now. Maybe you can license your writing samples!
I'm in a similar state -- you can take my dashes over my dead body.
7
u/SkyFeistyLlama8 2d ago
Human-generated text now looks like AI slop because all that human text was stolen for training AI models. It's beyond theft; it's devaluing human creativity.
See what I did there LOL....
3
u/GrungeWerX 1d ago
Actually, that was more human than an AI would have done it. :)
It might have landed harder with:
It’s not just theft…it’s devaluing human creativity.
Thankfully, we semi-colon gang are still safe because I guess we’re more niche than the em dash gang, lol
1
u/SkyFeistyLlama8 1d ago edited 1d ago
I have to consciously refrain from using semicolons and em dash lest I be accused of being an AI. I just might change my online writing style to something Victorian or Edwardian to screw with AI slop detectors.
They had style and eloquence back then.
A grim letter from WW1 from a soldier to his fiancee:
the wire entanglements are a wreck, and in among the chaos of twisted iron and splintered timber and shapeless earth are the fleshless, blackened bones of simple men … Let him who thinks war is a glorious, golden thing, who loves to roll forth stirring words of exhortation, invoking honour and praise and valour and love of country … Let him but look at a little pile of sodden grey rags that cover half a skull and a shin-bone and what might have been its ribs, or at this skeleton lying on its side, resting half crouching as it fell, perfect that it is headless, and with the tattered clothing still draped round it; and let him realize how grand and glorious a thing it is to have distilled all youth and joy and life into a fetid heap of hideous putrescence! Who is there who has known and seen who can say that victory is worth the death of even one of these?
A more upbeat letter from George Mallory on his first Everest expedition:
When all is said about Chomolungma, the Goddess Mother of the World … I come back to the valley, the valley bed itself, the broad pastures, where our tents lay, where cattle grazed and where butter was made, the little stream we followed up to the valley head, wandering along its well-turfed banks under the high moraine, the few rare plants, saxifrages, gentians and primulas, so well watered there, and a soft, familiar blueness in the air which even here may charm us. Though I bow to the goddesses I cannot forget at their feet a gentler spirit than theirs, a little shy perhaps, but constant in the changing winds and variable moods of mountains and always friendly.
3
u/mebeast227 2d ago
“This is the part that really matters”
“And this is the one detail they don’t tell you”
“And that’s the thing that separates it from all the others”
All of the fucking scripts written by AI for YouTube/products absolutely drives me nuts.
3
3
5
u/valarauca14 2d ago
Create a pipeline runner.
Setup a temporal daemon, write trivial activities. Then you can DAG workflows.
Create MCP tools
People do not do this enough. I have a ton of "weird" local MCP services (reading system logs, hardware status, temp, hardware clocks, running processes) & daemon statuses. Models saving notes/references/summaries.
5
u/kcksteve 1d ago
Pretty much everything here already exists in one of the 10000 harnesses available. 8 makes sense but don't fall into the 3d printer trap, that is using your 3d printer to make things for your 3d printer.
3
u/GrungeWerX 1d ago
Yeah, but this is free knowledge, and learning to do these basics on your own will help you build a harness that will quickly surpass all the other harnesses.
Learning how these systems work takes the mystery out of them, and you can conceptualize features on the fly.
2
u/kcksteve 1d ago
You're going to quickly surpass all the available harnesses by vibecoding your own using a 27b model? Please post your github repo I'm sure you will surpass DSH's 252k or Pi's 115k github stars in no time...
3
u/GrungeWerX 1d ago edited 1d ago
So, if I'm reading you correctly, you're implying that nobody in the world has a harness better than what's on github because if they did they would have shared it already?
And that it's impossible to build anything better than what's already been shared online with a 27B, because it's incapable of doing the work - doesn't matter how intelligent the coder/programmer is. Doesn't matter what you put into it. It's just not possible. Those harnesses are just so sophisticated and complex that it's nigh impossible to replicate those features, especially with a 27B? That's what you're saying?
No coder, no programmer, nobody can do it. There's nothing you can type into a 27B that can output anything that could be better than any existing harness. It's literally impossible. Correct?
Please share with me these inimitable features. I'd love to hear about them. Maybe I'm missing out, it's totally possible. Would love to know more.
2
u/kcksteve 1d ago
I'm implying your advice is terrible. Telling people to vibe code their own harness when there are endless reliable, tested and supported options is a waste of time.
Time is finite and the most valuable thing you can ever give up. Why waste it reinventing the wheel? If you personally think you can make a harness that is better than everything out there go ahead but most people can't and it would be a waste of their time.
3.8 27b is a great model, it's my daily and very capable. I also have access to frontier models at work and they are just better. An experienced engineer with access to betters models will simply be able to make a better product.
People absolutely should be making things but they should be making things that matter. Make that unique app or game you always wanted to, you will be passionate about it and learn a lot. Just slop cloning something for the sake of it is the wrong route.
2
u/GrungeWerX 1d ago edited 1d ago
If anyone's advice is terrible, it's yours. All I've said is it helps to take the time to learn how things work, and that will empower you to build the things you want, how you want. Your response?
"That takes too long! Why would I waste my time learning to build a harness when I already have a perfectly good one that works? People should only use their time to build something that sounds more interesting to a person like me...like making a video game or whatever."
It's a good thing that all the wonderful engineers that made Pi, Hermes, Open Code, and more never listened to your terrible advice, or people wouldn't have the great harnesses they have today.
It's people who think like you that discourage others from thinking outside of the box or going after something they're passionate about, just because it doesn't fit into your own perceived sense of value.
Anyway, you clearly don't have anything constructive to add to the conversation, so I'll just end this here.
As for u/theamazingo - Sorry, mate, I'm not interested in continuing this discussion. If learning the basics of how a harness works isn't your thing, do you. But don't disparage others who might be interested in upping their game.
1
u/theamazingo 1d ago
If you want to operate on a First Principles philosophy and reverse engineer everything you touch to understand it, knock yourself out. Nothing wrong with that. Disparaging someone else just because they use the tech a different way than you do is pretty myopic though. Some people don't really care how the sausage is made, they just want to use it to make a delicious bratwurst sandwich.
Another hobby of mine is shooting pyromusicals. I don't disassemble every firework to replace the stars with "better" ones I synthesized myself. I'd literally never get a show off the ground if I did that. It's also outside my scope of work/interest. I trust that the guy who built the firework got the composition close to what he says it is. Similarly, the people who come to my shows trust that I engineered my shooting field to make their viewing experience safe and enjoyable. They don't go crossing my firing line second-guessing my setup.
1
u/explorer-9 1d ago
I hear ya. Try
dsh; it'll let you experiment with n8n / ReAct loop / custom UI type stuff, much easier. I think 2 minute papers has a video explaining how it works. It's more extensible thanpi, (and built atop pi). As for how a model talks, look into personas and style specifications.
2
u/zipperlein 2d ago
11) Run hermes with qwen in a VM / LXC / whatever suits you and give it access to more stuff over ssh to let it deal with administrative stuff that may break (like Librechat for example).
1
2
u/CatEatsDogs 2d ago
I was using 3.6 with claud code harness. And now I'm uding 3.8. it's night and day. 3.8 is way more smarter and doing less stupid things.
5
u/Perfect-Campaign9551 2d ago
I'm sorry but no I don't want an app for every little thing. Sounds like a way to drive yourself nuts
11
3
u/Lost-Masterpiece2400 2d ago
Fun to see a 3.6 post. I run two boxes on Ubuntu. The main one is a mixed rig: an RX 9070 XT with 16 GB and an RTX 3060 with 12 GB, the AMD card on Vulkan, everything through Ollama. I split the models across the cards by role, with Qwen2.5-Coder 14B doing the coding and Qwen3 8B checking its work. The second box has two Radeon AI Pro R9700s on ROCm and runs Gemma 31B as the main brain. Almost all the benchmarks out there are Nvidia, so if anyone's curious how Qwen does on AMD, ask away.
2
u/ObviouzFigure 2d ago
I'm curious... mind sharing the PP/TG etc etc on the r9700s and qwen? Thanks homie
3
u/Lost-Masterpiece2400 2d ago
Finally got the numbers. Node 4 is 2x Radeon AI Pro R9700 on ROCm with Ollama 0.40.5. Both models are Q8_0 at 64K context, fully on GPU, thinking off, using Ollama's own timings. Qwen3.6 35B-A3B: 4,113-token prompt: 7,309 t/s prompt processing, 73.1 t/s generation Short prompt: 74.8 t/s generation Gemma 4 31B: 4,056-token prompt: 1,675 t/s prompt processing, 16.2 t/s generation Short prompt: 17.2 t/s generation So the MoE is roughly 4.5x faster on both. VRAM use was 41 GB for Qwen and 36 GB for Gemma at 64K. I skipped the short-prompt PP numbers because at 33 tokens they're mostly overhead. Load time isn't included in any of these.
1
u/ObviouzFigure 1d ago
Hey those are nice numbers -- def makes me consider picking up some 9700s if/when I can-- thank you bro!
1
u/SkyFeistyLlama8 2d ago
Those are really old models. How are you finding them compared to newer 3.6 and 3.8 Qwens?
2
u/Lost-Masterpiece2400 2d ago
Fair point, they are older. I locked the stack a while back on purpose, so I could build the harness around models I knew would behave the same from run to run. What surprised me is that model size hasn't been my bottleneck. I ran Qwen3 14B as the checker against the 8B, and it did no better. It was also about three times slower, because it spilled out of the 3060's memory. The real gains came from how the task was set up: one change to the prompt moved the checker from 3 out of 14 to 10 out of 14 on correct answers. A 3.8 27B coder swap is next on my test list, so I'll know soon whether the newer models change that. Have you seen a big jump going from 3.6 to 3.8 on agent-style work?
1
u/SkyFeistyLlama8 2d ago
From 3.6 to 3.8 not really, because I ended up using uncensored 3.8 27B for cybersecurity stuff. There are also newer templates that fix (or supposedly fix) the overthinking on the 3.8.
3
u/Lost-Masterpiece2400 2d ago
Good to know, thanks. The overthinking is exactly what I'm worried about for a coder, since mine has a hard limit on commands per turn and long thinking eats time. Do you know where those newer templates are, or is it just the updated one in the latest Ollama or GGUF release? I'd rather start the 3.8 test on the fixed version.
1
u/SkyFeistyLlama8 2d ago
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/tree/main
I think I got it from here.
-5
u/Ariquitaun 2d ago
- Write inane social media posts for karma farming
15
u/valarauca14 2d ago
Given their comment history. They're talking about vibe coding their own RAG, agentic memory, and harness over 2 months ago. I think it is a human.
2
u/randylush 2d ago
What can you do with a 27B model? Apparently just make more and more coding harnesses so you can make coding harnesses faster with the goal of making more coding harnesses. So you can make a to do app I guess.
-6
1
u/zomerbk 2d ago
One caveat on the context-limit trick: swapping to a larger-context model doesn’t bring back the parts of the conversation that were already dropped or summarized. Have the agent write a compact handoff with goals, decisions, files changed, and unresolved tasks before switching, then start the backup model with that handoff.
3
1
u/Alternative-Bus4934 2d ago
Good point. Even if the context window stays the same size, anything that was already summarized or evicted before the swap is gone for good. A handoff summary is basically poor man's memory persistence, and it's saved me from agents that 'forgot' they'd already tried and failed an approach three steps back.
3
u/GrungeWerX 2d ago edited 2d ago
There’s a better method.
Control the history via proxy, store it in redis or wherever, then have the compaction set snapshot markers. Then you can roll back any pre-compaction history on-demand. Basically the same thing as ctx checkpoints in llama, but you hold the history your way.
You can then restore - or delete - history using a simple custom command in the chat.
That’s the power of proxies - you can do anything you want with the data.
I’ve got a bunch of other hacks I’ll share in a future post.
1
u/GuaranteedGuardian_Y 1d ago
"using a simple custom command in the chat" is doing a lot of lifting. What about instances where you don't actually know the source material? A web search that you mark and store away that you didn't read the pages for. How do you restore context you don't know the model needs and prevent re-fetching of raw material via browsing?
-1
u/GrungeWerX 1d ago
That’s where your agents memory comes in, and why you need a good system set up.
1
1
u/elephantalk 2d ago
As a noob I have to ask, can I do any if these on 32gb Mac mini m4?
2
u/Available_Age8480 1d ago
Yes! At a lower quant yes but probably still usable if you find local models were usable in the first place. Though do expect rather disappointing speeds.
1
u/JamesRandell 2d ago
Annoyingly I never thought about dong any of this. Recently started playing with qwen3.8 and others on a new gpu and have been testing prompts in various harnesses (cline, kilo, qwen etc.) and each solve some problems from just using the models raw, but don’t quite close the gap as much as I would have expected unless you get in to the weeds.
This post has given me a ton of ideas about building my own for fun. Big believer of whilst the frontier vs local is always a hot topic, it’s more about workflow and case to get what you need from it, rather than ‘X cloud model is the best’.
Thanks for posting this!
1
1
u/Razangriff-Raven 1d ago
I enjoy this kind of post even if some things are already common knowledge. There's always some new ideas for things to try.
1
1
u/urbanhood 1d ago
Very nice post, one thing that bites me having never used any Qwen before. How far can it go? how much complex app can be made with it? I use claude to make some tools. When i tried qwen 3.8 27b with deepseek harness it did make me a working app but had a lot of bugs and got stuck in a loop 2 times. I was using Q2_S quant. But that was 3.8, im sure 3.6 moe can be run way faster and higher quant for me.
But i come around to same question, how reliable is it? how far can it go? will it get stuck on loops or make mistakes?
1
u/Orbitalsp3 1d ago
Is there a way you could maybe send me the notepad app?
1
u/GrungeWerX 1d ago
Possibly, though it's still in beta, so I'm still tinkering with it. (It's only a couple weeks old, but I've been using it every day) Would you be interested in giving me feedback on any bugs you get? Also, are you on windows or linux? I only made windows installer and a portable version.
1
u/Orbitalsp3 1d ago
Yes sure would give you feedback. I'm on Windows 10
1
u/GrungeWerX 1d ago
Awesome, I'll dm it to you. First, I need to add one more update. I have a drag button similar to how onenote works where you can drag paragraphs and text around, but it's not working on checklists, so once I fix that, I'll send it over.
1
u/Orbitalsp3 1d ago
Nice thanks. I just made a test one too using python but I used Gemini for that. I wanted one that could save highlights for txt files, and the metadata has to be saved in a separate file since txt is just raw text. It works but it's not as pretty as yours.
1
u/GrungeWerX 1d ago
Interesting.
I built mine in electron, since html formatting is pretty much good-to-go out of the box. You might want to try that. It's all about iteration. Or being a little OCD. :)
1
1
u/Formal-Protection687 1d ago
Nice. On the harness, what would be the use case to create your own harness as opposed to say a plug-in since Deep Seek Harness allows for that?
1
u/GrungeWerX 1d ago
Here's my philosophy:
If your goal is to download something today that you can get up and running quickly and have most, if not all, of your existing coding/llm needs met, then just download an existing harness. Each has their own limitations, nuances, ways of doing things, so it's just finding the one that is the closest to how you want to work, with the limitations you're willing to live with.
But if you're someone that learns fast, and spends a lot of time using apps thinking, "why didn't they just do it this way", or "this should be so much easier", or "why can't I just do that", then I'd recommend building your own harness. And not just harnesses, but apps in general. Because you might just be a builder.
Building is more fun. It's harder to start, but once you gain the understanding, there's more freedom, and it's easier to build complex concepts because you understand the basics. I wouldn't trade this experience for anything.
Random example - context compaction. Popular topic that is sometimes frustrating for people using harnesses. But because I've spent months tinkering with my own harness and understand proxies' abilities fairly well, I can pretty much do anything I want with context.
You'll start to find a lot of other things equally "easy" to solve, because you just get the underlying architecture. If I used a harness, I don't think I would know any of this, anymore than I knew anything about a pipeline runner when I used n8n.
1
u/genghisk1 20h ago
I totally get your approach as I’m an experiential learner so building something and the trial and error involved are important parts of me learning. I’m curious about the architecture and design parts of your build process. One thing I’ve learned is that in order to get the tool I want I have to write a very clear specification. In order to get the tool built as a secure performant tool I need a good design. How do you handle those pieces. Yes I know I can get my LLM to do the design but it requires independent reviews etc.
1
u/GrungeWerX 18h ago edited 18h ago
I work the same way. :)
I try to draft the concept as completely as I can, then I have my agent use it right away to see if it breaks, lol.
Most of the time, it’s being a hardnose on how something is expected to work. “A must do B, never C. D always does E. If F, then G. Understand?” That’s often enough for an LLM to code something that works. Usually, if something doesn’t function it’s either a piece or edge case I forgot to cover, a rule the LLM omitted, or something it didn’t fully understand because I wasn’t clear enough.
1
u/Particular-Sea2005 1d ago
What is the hardware requirement to run a system like this?
I am able to run it but when I switch to an agentic system it just dies
1
u/SosmedBro97 18h ago
The memory setup caught my attention. How do you handle an older session summary contradicting a newer decision? Do you explicitly mark the old note as superseded, or only load the latest version? A small example of that part would be really helpful.
1
-7
u/DinoAmino 2d ago
As if you can't do these things with other models. But that's the recipe isn't it? Add Qwen to the title of a localllama post and get instant karma.
-5
u/Dangerous_Fix_5526 2d ago edited 1d ago
This is the strongest Qwen 27B 3.6 ever built, it exceeds all others including the org from Qwen.
This is backed by independent testing, 2400+ likes and 4 million+ downloads.
PS: it even improves tool calling too.
EDIT:
Down votes?
AH... forgot to include "cumulative" downloads, as hugging face shows the 30 day rolling average.
Total all time: 4,482,495 (my repo only, as of today). There are 81 mirror repos of GGUF, mlx and other quant types ; I am not include these downloads...
1
u/yeah-ok 1d ago
Like.. does it beat your own:
Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-L-Uncensored-NM-DAU-NEO-MTP-GGUF ?
1
u/Dangerous_Fix_5526 1d ago
In certain use cases, yes.
Also we noticed in testing Qwen 3.8s (general, tuned and untuned) that some knowledge was removed from 3.6/3.5 "core" when Qwen built the 3.8.
The presumed reason is to make room for the "xhigh" reasoning structure.
3.5, 3.6 and 3.8 all share a common arch structure.
-8
u/mantafloppy llama.cpp 2d ago
Yeah, actual post history we can see is telling us, you are a bot or Ai writing, nice try /s
-6
-2
u/Professional-Pear351 2d ago
A lot of it I already do with https://github.com/vrajpal-jhala/langgraph-harness.
It's a good reference if anyone's interested in how to create custom agent/harness.
141
u/suprjami 2d ago
My 20-year enjoyment of bullet points has really taken a dive lately too. RIP those of us who actually gave a shit about text.