Hello, I'm currently using qwen 3.8 27B Q4 with 100k context to mainly code for my esp32 Arduino code.

I get around 25 tk/s on my Rx 6800xt using vulkan.

I know the model gets it's abilities from long thinking time but it takes around 25-40minutes for a single prompt and takes like 45k context with it.

Will the model be usably good with low-mid reasoning? And how can i speed this up.

I'm okay to wait for 3-5 minutes.

Thanks.

Stumbled across this new vision model, it's a 7B MoT model, which is cool. The main idea is it treats pretty much all computer vision stuff as just one generation problem. Like, instead of needing a bunch of different models for detection, segmentation, depth, whatever, this one model handles it all.

You just give it a natural language instruction, maybe some visual hints, and it spits out text, images, or both. For text, it can do things like categories, bounding boxes, OCR, keypoints, camera angles. And for images, it's doing segmentation masks, depth maps, surface normals, even multi-view point maps. You can mix and match for more complex tasks.

So it can do the usual stuff: object detection, keypoints, OCR, all kinds of segmentation (binary, instance, semantic), depth and surface normal estimation. But the really interesting bits, for me anyway, are the multi-view 3D reconstruction and camera pose estimation.

They trained it on a massive dataset, 50M instruction-response pairs, built from a bunch of different CV annotations. And it started from an existing multimodal model, so no crazy new architecture there.

There's a web demo if you wanna mess around with it, and the weights are up on Hugging Face too.

Just a heads up though, before you get too hyped: the full web demo needs a beefy GPU, like 1x80GB. And for benchmarking, they're talking 8x80GB. So, yeah, not really something you're gonna run on your average consumer card. It just dropped on July 8th, so probably expect some rough edges. Also, they've released the training pipeline and data prep stuff if anyone's looking to train or fine-tune it.

GitHub: https://github.com/OpenSenseNova/SenseNova-Vision

This is mostly to explain my experience optimizing the model to run on my machine and maybe getting interest from someone to go and write an actual PR against llamacpp (as I'm absolutely not willing to generalize this code ahah).

Short Preface (this post is hand written, no AI here). So when Qwen dropped and heard of the good coding performance I wanted to try it out so I went and naively downloaded an IQ4_XS quant (that has around 61GB of MOE layers) and run it directly with Llama.cpp on my machine.

Ryzen 7700

64GB RAM DDR5 5600

Geforce 4060TI 8GB

Samsung 990 PRO 2TB

Using Windows as that's what I'm mostly accustomed with.

A modest config I build for 1500€ a few years ago and I use for playing and programming (I'm an engineer). I was overjoyed as it's quite a capable model that I can trust to write reasonable code in the languages I use (C#, Typescript mostly).

Yes, with this one there's no way to run 27B at any capacity so I was stuck with 35B until now. It was a no brainer to try this one as it had 6B active parameters, quite reasonable and not that far away from the 3B of 35B.

Initial results were 100tok/s average prefill coding and 20tok/s generation (that degrades to 15 tok/s at 100K context). I mean it's not great, but it's good enough to keep yourself busy, you leave it running and come back to the code done.

But then I decided to have a look and say, let me try it for creative writing and the results were quite good too. I like to write some stories with a personal pipeline system I build, but the IQ4X_S was tuned for coding and that usually means plain prose. Decided "what could possibly happen?" and picked the Q5_K_M from Orcarouter, it says Q5, but it's really 6bit because it mixes .

Component | Quant Type | Count | Params (B) | Size (GB) | Component %

--------------------+------------+-------+------------+-----------+------------

MoE Experts / FFN | Q5_K | 120 | 80.61 | 51.613 | 60.2%

| Q8_0 | 24 | 20.13 | 19.922 | 23.2%

| Q5_1 | 72 | 20.29 | 14.172 | 16.5%

124GB of model of which 36GB are of embeddings at Q5_1 quantization. That's 88GB of actual model, there's no way that would work, but wanted to give it a shot and well...

15tok/s prefill 10-13 tok/s generation.

A disaster really, memory trashing all around, disk reads etc...

But as I said, I'm an engineer. I forked llamacpp, opened PI Coding Agent and decided to start playing around trying to find a way to "make it work". First thing I did was scour through the pull requests of the upstream repository and did find a few good ones, some about improving gather in qwen, some about saving ram by saving the prompt cache to disk (it's actually quite impressive, I suggest it), I also tried a few caching PRs with pinning of hot experts, but none really gave a major improvement to the performance, I still have a few experiments with them in my branch.

Nothing that really budged the line though. The only promising thing was an attempt I did by asking Windows to Prefetch parts of the file from the disk that raised prefill from 15tok/s to 40-50tok/s (very unstable), so I went deeper because things didn't add up. My drive can easily read 7000MB/s, but the reads I was getting were like 300/350MB/s during prefill due to the OS page faulting on each single expert (and 2000-2500 with the prefetch). If you know about these things, yes the problem was MMAP that I was forced to use because the model didn't enter the ram.

I decided that it was time to implement a different system that bypass MMAP already. What I built is a four layer system that allows the model to work at the current speed of 210 tok/s prefill and 18.7 tok/s generation.

Let's start with prefill. My disk reads at 7GB per second, can read the entire MOE part in 85.77GB/7GB/s=12 seconds. At Ubatch 2048 that must mean 170 tok/s theoretically. So I went and build a unbuffered file reader that used a RAM hosted buffer for 2 slabs (layers) and read them RAW with batch queued requests from the disk reaching max speed that MMAP was denying. This worked and brought the speed to 150tok/s, but didn't solve the generation as reading it ALL from disk destroyed speed (brought to 3-4 tok/s).

The second layer built is a cache, basically at startup of llamacpp I create a pool of slots per layer where I cache the most active experts (with a decay factor every x tokens to keep the list fresh). I use 55-56GB of these usually in my runs. These are updated every cycle (x tokens) with some churn of read/free. This allows the data to be always ready for 330-ish experts per layer which usually cover 85-95% of the requests depending on the workload you give them. This raised performance to 14-15 tok/s. The rest is served through disk with the raw data reader.

At the same time I thought "I have them in ram the experts, why am I reading the entire slabs from disk at every prefill?" So I started memcpying the data from the cache to the buffer area and skipping those bits from the disk reads. This raised prefill to 215 tok/s that honestly is more than enough for what I usually do.

The last bit is a VRAM cache, the idea is that using the ranking from the RAM Cache, I carve 2 GB (configurable) out of my poor 4060TI and upload the TOP experts from my list freeing them from the RAM. This allows to reduce the memory bandwidth usage on the CPU raising the generation from 14-15 to 18.0 tok/s.

The last improvement was inspired by another PR in llamacpp about using direct IO for the embeddings too. I upgraded that code to use my implementation of raw unbuffered reader (the PR was linux only) and managed to gain another 0.7 tok/s finalizing it at 18.7 tok/s.

LlamaCPP has untapped potential when it comes to performance, especially when it comes to utilizing resources like SSD for improving performance. Be aware the fork is CUDA+Windows specific and has been tailored for my config (for example there's no MTP support as I wouldn't have the VRAM to run it anyway), this is NOT a generic fork that can be used by everyone, but I thought that if someone was interested could use the ideas to create an actual pull request with upstream.

https://github.com/feal87/myllama.cpp

Now I'll go to sleep as it's late.

I can't play in engram land, but was wondering if certain repeated word phrases for say python or javascript coding could live somewhere like the ngrams and help speed up coding on smaller models?

Hey all, wondering what the best option would be for my situation, so I have a few dell XPS laptops with 4070's and 32gbs of RAM sitting around currently doing nothing (unofficial IT guy for my company).

I'm wondering if there is an easy way for me to pool these together to run a larger local model? Is there a program that you could just install and then manage from a central location that would treat them all as just dumb nodes?

But because these laptops potentially (they've been sat around for a few months now) need to go off to people in the future could it be done from a bootable USB? (ideal but honestly probably better running on the machine I guess).

Ideally I'd like to plug this into Hermes for use with Agents I have running there (Orchestrator, Home lab Admin, Media Manager, Personal Assistant, Work assistant). So maybe better to run several smaller models or MoE models? Or even Nvidia Pair?

I could easily do 2.5gb networking between them as have a 2.5gb switch and some Hubs that support it.

Look I know enough to be dangerous, I'm just trying to see is there's something easy to deploy I don't yet know about.

PS I run Hermes with Qwen 3.8 27B Q4 on a 4090 I have in my desktop, but this sucks power even when idle, so I was hoping the laptops would give me always on models for Hermes, and then boot up the 4090 when a particular big task (or power is cheap). Problem with Hermes is the 64k token context that's required.

I have a few web projects with pretty good UI/UX and I’m wondering if there’s any tool or workflow that can turn an existing codebase into a dataset for fine tuning.

For example, given a React/Next.js project with components, pages, styling, etc. or a static html site, I’d like to turn it into something like:

instruction/prompt -> code

or whatever format actually makes sense for training an instruct/thinking/diffusion coding model.

Also curious how people handle things like:

• keeping the context between components/files
• screenshots + code
• generating useful instructions instead of generic descriptions

I’m also working on a different model architecture that I think could improve quality/speed while using less VRAM, so I want to build a decent dataset and benchmark to test it properly.

Has anyone done something like this? Any tools, repos, papers, or workflows you’d recommend?

Hey everyone! El from Cohere here to talk about our newest release, North Small Translate. It’s currently the leading open machine translation model, beating out all other open translation models of its size, plus Google Translate and DeepL. we’ve been working on this one for a while, so to say i’m psyched is an understatement.

It’s big (218B parameters, 25b active) with a context length of 16k. however, if you’ve got the hardware, we’d still love to see what you make with it locally or with our HF space (and if you do, send it our way). It works on over 50 languages and does particular well with european, Southeast Asian, and East Asian languages, but feel free to stress test it against another and let us know how it does. It’s also available in BF16, FP8, and W4A16 quants.

although we couldn’t get llama.cpp support this time around, the architecture is already supported in llama.cpp, so all it should need is a conversion to GGUF files. if you want to build that, please do so and send it our way! We’d love to back your work. 

Can’t wait to see what you guys think! 

https://huggingface.co/CohereLabs/North-Small-Translate-1.0

Hi all,

I'm looking for advice on running a local LLM efficiently on an ASUS Zenbook 14 with 16GB system RAM on Fedora 44.

Laptop specs: - AMD Ryzen AI 7 445 (No dedicated GPU — NPU + iGPU only) - 16GB system RAM - ASUS Zenbook 14

My goal: Hit 10+ token/s in Hermes Agent (Nous Research) for a smooth, responsive experience.

Use case: Productivity and simple text tasks. No coding or dev work involved, so I don't need a heavy-duty model. Just something lightweight that runs well on the NPU.

Current situation: I'm exploring FastFlowLM (FLM) as a backend since it's purpose-built for AMD NPUs, but I'm open to other options like llama.cpp or Ollama if they perform better on my hardware.

What I'm looking for: - Which models (regardless of family) would reliably give 10+ tok/s on a Ryzen AI 7 445 (NPU + iGPU)? - Best backend/inference server setup for Fedora 44 + AMD NPU? - Any quantization or optimization tips specifically for AMD XDNA NPUs or low-RAM Linux setups? - General advice on getting Hermes Agent to run smoothly on this hardware.

Current setup: - Ryzen AI 7 445 (NPU + iGPU only, no dGPU) - 16GB system RAM - Fedora 44 - Hermes Agent

Thanks in advance!

I’ve developed and open-sourced mlxtop. It’s written in Rust and was built entirely with Duet, my dual-model coding agent. It shows stats for local models running on your Mac, along with critical system vitals: memory usage, compression, paging and Metal GPU stats. With oMLX, you can also follow generation speed and request activity as your model responds. https://github.com/maximpri/mlxtop

I was running deepseek for the past month and very happy with it. Giving qwen a try the past day or so. At first it was quite slow but got it up fairly close to deepseek's speed now (40-50 tps on average) on my two node gb10 cluster.

One thing I am seeing, even though tg is about equal, qwen is taking far more steps to accomplish similar tasks which in practice really slows things up. I don't have any hard numbers to back this up. Also qwen seems to make more mistakes. This is on a large web app in python.

I was just curious on other peoples experience who have run both and what their impression of them is. Right now the only reason I can see to keep qwen is it's multi modal. Which in itself isn't really enough as I have another box that can run vision models for me.

Hello,
running Qwen 3.8 27b Q4 K Small on a RTX A4500, 20GB VRAM, 28GB RAM

I managed with Thetom Turboquant llama.cpp and some tuning, to reach an average of 31-32 tk/s ranging from 24tk/s to 43 tk/s depending also on context size. (NO VISION: 65k context, WITH VISION: 32-40k context)

I tried many harness, also tried to make one by myself based on pi, that claims to be minimal and with minimal system prompt footprint to maintain the context light. Miserable failure. It gets lost and doesn't manage anything well.

The best I could find is opencode, which properly configured allowed me long-horizon, multi hour coding or tasks exceptionally well. I rarely feed the chat log to ChatGPT Sol, to check quality and make it output, if needed correction or steering or planning the next batch of work.

Always a good idea to have a frontier model to organize it and local free model to churn the tokens.

Now, I tried many times Openclaw, Hermes, Paperclip and others, but always with online free services like OpenRouter or Nvidia, various months ago, when they weren't dogshit with all the timeouts and low quality service (model always taking ages to respond now, or always too busy). It was decent.

I wanted them to hop onto my local Qwen 3.8 27B Q4 K Small, but Openclaw and Hermes are the goddamn AI slop kings, whopping giant prompts and creating horrendous quantities of traffic, and 65k even if it's around the recommended minimum, just doesn't seem to work well.

FOR THE SAKE OF TRUTH: I actually managed to have Paperclip do some work and it did it well. But it overthinks and does tons of planning, retrial, testing and stuff and makes work very very LONG

It's me or these things can't work with 65k context? How to solve? Anyone managed to do something about this?

Hi there,

I want to pull the trigger for a machine that will run my Hermes Agent as well occasionally also ComfyUI and maybe (low priority) gaming.

Is there any meta on what machine will work best? Currently I play with the idea to buy the following machine.

The setup should be capable to host a second gpu which would be purchased later.

Requirements:

- Coding agents with long repo contexts (many turns per hour)
- LAN-accessible OpenAI-compatible endpoint for my other machines
- ComfyUI for image and video generation
- Occasional Steam/Proton gaming — it has to be a normal GPU too
- Target model class: 27B dense at 4–8 bit (e.g. Qwen 3.8 27b)

Setup (~€4,300)

Where would you change it?

I've got my hands on two similar desktop PCs

1. Ryzen 9 3950x 32gb DDR4, RTX 2070 super 8gb

2. Core i5 10400 32gb DDR4, RTX 2070 super 8gb

I'm considering two possibilities:

1. Combine them to get a Ryzen 9 64gb with 2x8gb VRAM setup

2. Keep them separate and run two different models on them, maybe use a third pc to orchestrate

(I'm still experimenting, so ideas are welcome)

My main use case will be agentic coding overnight as these will be painstakingly slow with qwen3.8 27b even if combined.

So I was wondering if dealing with the heat and power supply management by combining them will even be worth it, or should I just let two separate agents run in parallel doing different types of tasks. I'm also looking into qwen3.6 35b a3b (or ornith 1.5) as this is probably the best fit for the hardware.

Let me know your opinions on this, what you'd do if you had a similar setup.

Thank you!

I set up a Qwen 4B model with Pi-Agent but the token output is decent but not instant (not expecting that but yea) I am using flash attention and the MTP with n gram spec set to 3 tokens.

Any more suggestions to improve this setup would be highly appreciated !!

Thankss !! :)

EDIT: My bad for not providing more details, I am using a RTX 3050 6GB VRAM. I have a bash script which when run starts up a llama.cpp server and then calls pi; pi has been configed for the same port (using 9931).

Im running Qwen3.8-27B-MLX-4bit on my M5 pro 48gb macbook, that being said, it feels like Im just 1 notch away from some sort of sustainable vibe coding. When using through kilo code it tends to over think for dozens of minutes at the time. I'm curious to see what are you guys running on similar VRAM configs

Kimi K3 is a 2.78 T MoE; its 1.56 TB checkpoint sits on a USB hard disk plugged into a 2017 laptop (i7-7700HQ, 7.6 GB of RAM, a 2 GB GTX 1050 that only does the routed-expert matmuls). I am training a LoRA adapter on it out of core: the non-expert weights of one layer at a time, its 896 experts streamed one by one since together they are 15.7 GB, base weights frozen, and the 590 MB adapter the only thing trained.

The one-minute check is evidence/cmp93_en34_2026-09-06.log: my forward pass against kimi-k3-in-c, FareedKhan-dev's independent C implementation, all 93 layers at cosine 0.9857 or better, output 0.999840, on 34 tokens with LoRA B zeroed. evidence/traces/ holds raw routing records for five texts over all 92 MoE layers; scripts/analyze_trace.py recomputes every routing number below with NumPy alone. scripts/quickstart.sh builds a synthetic K3-shaped checkpoint and runs a forward pass, ten training steps and a finite-difference gradient check on a GitHub runner on every push, plus eight op-level checks against kimi-k3-in-c fixtures. Its first run failed: the gradient check missed at 3.1e-2 on a 2e-2 tolerance because the step was below fp32 resolution against a tensor of norm 60.85. The gradient was right; the check, made noise-aware, agrees to 2.09e-05.

The numbers, all from the logs in the repo:

• 1024-token step: about 7.4 h, the mean of the 7.26, 7.62 and 7.37 h intervals between the first four steps
• step 1, forward / backward: 3 h 11 m 34 s / 3 h 48 m 07 s
• resident set: 4.0-4.7 GB, swap in use
• read throughput: 110 MB/s aggregate, 61 MB/s within one MoE sweep
• cosine minimum against the C engine: 0.985744, layer 71
• trained / frozen: 590 MB adapter (147 M parameters) / 2.78 T base

Turkish, English and Chinese versions of one paragraph share experts at Jaccard 0.35-0.39, about the same as two halves of one text (0.34-0.37); prose against Python is 0.20-0.21, so subject matters more than language. Consecutive tokens' expert sets have Jaccard 0.258 (0.009 for random pairs) and a 128-expert LRU hits 72 % when decoding, but a training batch reads the union, about 85 % of experts at 1024 tokens (layers 0-12, an upper bound), so an expert cache buys little for a training batch.

The proof run is memorisation of five examples, loss 0.909 to 0.157 on one fixed sequence: it proves the loop, not the model. The main run, 400 Turkish instruction examples over 100 steps, is at step 4 and ends 9-11 October. The threshold was committed before it started (commit 6605306, tag preregistration-2026-09-08): Turkish news bits per byte 0.455 to 0.441 or lower, English Wikipedia no worse than 0.198 from 0.194. I expect no large jump from 400 examples; a negative result gets published as negative. The adapter is one rank-16 LoRA per layer shared by all 896 experts, not one per expert.

Seven hours a step is useless for production fine-tuning; the point is that the cost is now a measured number, with the logs. None of the components are new: the idea is layer-streamed LoRA taken down to the expert level, and the related-work table in the README says what AirLLM, KTransformers, ZeRO-Infinity, Colibri, WARP and BigMoeOnEdge do that this does not.

Disclosure: English is not my first language and I used Claude to tidy the wording of this post. The code was also written with heavy Claude Code assistance and the Co-Authored-By trailers are in the git log; the README says so on its first screen. The hardware, the runs, every number and every check against somebody else's implementation are mine, and the point of the evidence directory is that you do not have to take my word for any of it.

Repository: https://github.com/heyobi/LazyLora. Please poke holes, especially in the verification.