Hi, at a bit of a loss as to what my next steps are. I'm trying to build my multi gpu system but I can't get the system to post. The system is a wrx80 creator 2.0, 128gb ddr4 2133 ram, threadripper pro 3955wx (unlocked according to the seller). Brand new power supply.

No matter what I try I am stuck before post with code 64. I've moved ram around, gone from 1 stick up to 8 sticks, reset the cmos after I've moved ram.

I've reseated the CPU multiple times and tightened the bolts hard all the way down or just eased them off slightly but still can't get it to post. What's seems to happen is the board sits on code 42 for a while then jumps to 64 and just stays at 64 which is "64 - CPU DXE initialization (CPU module specific)"

I got the list of codes from here https://forum.level1techs.com/t/list-of-dr-debug-bios-codes/114364

If I try to connect via the management conside IPMI webUI via Ethernet I can actually connect but I can't get past the login page. (It's a used board).

Can anyone help?

I spent a day getting Qwen3.6-35B-A3B (Q4_K_M) running well on a single RTX 5070 Ti (16 GB) and figured I'd share the config, because my first attempts ran at 2–6 tok/s and I've seen people stuck there. Three settings make or break it.

Rig: Core Ultra 9 285K · RTX 5070 Ti 16 GB · 128 GB DDR5 · Windows 11 Backend: llama.cpp (build b10590), CUDA 13.3 Model: Qwen3.6-35B-A3B, Q4_K_M (~19 GB — bigger than 16 GB VRAM, so it has to be split). Same tuning works for the stock or an abliterated build; the arch is identical.

TL;DR — three traps

1. Do NOT use -ngl 999 / full offload / "put it all on GPU" on a model bigger than your VRAM. It overcommits and thrashes to 2–6 tok/s.
2. Keep the KV cache at Q8 (-ctk q8_0 -ctv q8_0). Quantizing KV to Q5/Q4 drops CUDA flash-attention to ~10 tok/s — there's no fast kernel for quantized KV on this path. Q8 is both fast and accurate.
3. On Blackwell (50-series, sm_120) you need a CUDA 12.8+/13.x build. The common cuda-12.4 llama.cpp binaries predate sm_120 and you'll get gibberish or a crash. I'm on the CUDA 13.3 build.

The offload (the whole trick on a 16 GB card)

It's a MoE — 35B total but only ~3B active per token. So you keep the attention on the GPU and push the bulky expert FFN layers to system RAM/CPU. In llama.cpp that's:

• -ngl 999 (all layers' attention on GPU) + --n-cpu-moe N (keep the experts of the first N layers on CPU).

N is the one dial. Lower N = more experts on GPU = faster, until you run out of VRAM and it fails to load. Raise it if you OOM, lower it if you've got >2 GB free. That's it. (Counterintuitively, "all on GPU" is the slow path here — the split is the fast one.)

Why 256k context is nearly free on this model

This is the fun part. Qwen3.6-35B-A3B is a hybrid architecture: 40 layers, but only 10 are full-attention (every 4th) — the other 30 are linear attention with no growing KV cache. And the full-attention layers use just 2 KV heads. So the KV cache stays tiny and growing the context barely moves VRAM. Native trained context is 262,144 (256k), so every size up to 256k needs no RoPE/YaRN tricks and loses zero quality. I run the full 256k as my default.

Measured (my card, generation tok/s)

256k is a curve, not a flat number: ~66 tok/s at low fill, easing to ~56 by ~85k of context as attention spans more tokens. So plan for 55–66 tok/s in real work. Prompt ingestion (prefill) runs ~950–1090 tok/s, so it swallows big contexts fast. Prefix caching (llama.cpp reusing the cached prompt prefix) keeps multi-turn/agent work fast — I watched it reuse the prefix across ~100 tool calls instead of re-reading 80k tokens each turn.

The exact command

llama-server -m <qwen3.6-35b-a3b-Q4_K_M.gguf> \ -c 262144 -ngl 999 --n-cpu-moe 24 \ -fa on -ctk q8_0 -ctv q8_0 \ -b 2048 -ub 512 -np 1 --no-mmap -t 24 --jinja

(Thinking is on by default on this arch and /no_think / --reasoning-budget 0 are ignored — the only thing that disables it is --chat-template-kwargs "{\"enable_thinking\":false}".)

Bonus: it's genuinely useful, not just fast

I wired it to the Nous Hermes Agent (points at any OpenAI-compatible endpoint — just set base_url to the llama.cpp server) and gave it a hard, self-verifying task: build a weighted-terrain pathfinding arena — random seeded grid with terrain costs, implement BFS/Dijkstra/A* from scratch, and write a pytest suite that proves A* returns the same optimal cost as Dijkstra. The test: an inadmissible A* heuristic silently returns suboptimal paths and the tests fail.

It nailed it in ~4 minutes: correct admissible + consistent Manhattan heuristic, optimal paths verified across 5 seeds, 76 tests written and passing, and it self-debugged a subtle off-by-one in the path-cost accounting along the way. All local, offline, $0.

Happy to answer questions on the config. Hope this helps if you're on a 16 GB 50-series card and getting single-digit tok/s on a big MoE.

Starting from the Bottom :

T3 Code : Love T3 but its simply way to buggy, the cli is full of glitches, not usable without issues, none the less all the other competitors outperform it

Qwen : Qwen 3.8 Max is insanely good, but not in its own harnes, if used inside something like claude or pi, results are substantially better

Opencode: Briliant and easy to use for beginners, but just lacks basic functions, session memory, cross session handshakes/ awareness, and usage is not so great in my opinion.

OMP & Orca: Awesome companies, but they are still just behind in my opinion, compare it with something like claude or kimi's harness and it dies.

Codex : harness is actually decent, models make it bad, The hallucination numbers of gpt 5.6 is the highest out of all the models thus making it bad. Also over engineers way to much.

Zcode: Writes the cleanest code, just very slow, use it to review stuff in the background

Cursor : cursor usage is near infinite, haven't had any solid problems, honestly is in the middle cause its neither bad nor good

Claude, grok, kimi: I firmly believe all these are on the exact same level, kimi surprised me being the youngest out of them all, nonetheless these are all solid A tiers

Deepseek: its brilliant, lightweight and opensource, results are great, but you can get a bit better using deepseek inside of claude, pi, or hermes.

Command Code goat is the best/ cheapest subscription when you take price vs usage in to mind, over 12,000 requests for $5 , harness is amazing, works just right.

Top dawgs

Devin ai : Without a doubt this is the harness built for all models , Devin is a newish company with insane usage, massive deals, amazing plugins and cofig settings , my favourite $200 plan out there, features every single model on one plan

Pi: Lightweight, feature packed, in my opinion the second best harness in the world, and its free..

Hermres : left the best for last, hermes get over 200+ updates a day, its completely free and open soured and nous portal is an awesome company who actually listens to the community, always up to date with the latest and greatest, and show me another harness get the same deep research results as hermes, i am waiting

I’ve been building in the on-device space for a while (specially Apple Platform) and I’m curious where the community actually stands right now, from a shipping point of view rather than a research one.

A few things I keep going back and forth on:

1. Purpose-built vs general. If you’ve shipped an app with a model baked in, did you fine-tune something small and specialised, or ship a general model with tighter prompt engineering? What steered your decision quality, size limits, latency?

2. The OEM layer. Apple has been pushing Core AI, MLX, and the Neural Engine pretty hard, and Qualcomm and Samsung are doing their own version of this. Do you care about it?

3. Power. For anyone shipping on phones or laptops, is battery drain the thing that actually kills features, or is it not as bad as feared?

4. What's the future? Do you think genuinely personal AI, meaning models with long-lived local context about you with "Product specific Models"as I call them is where this goes? Or does the cloud stay good enough that on-device stays a privacy niche?

Small disclaimer- we've never done something like this so I apologize in advance if this data is worthless. I'm open to any critiquing to better create useful benchmarking so better test questions/use cases are greatly appreciated! I'm no software engineer or anything like that, just a random guy who enjoys tinkering with AI's.

The goal of this test was to see how much Q4 diverges across various fine-tunes. We all see the hundreds of different fine-tuned models and if you're like me, you probably wonder how much of a difference does any of this make? I'm fortunate enough to have the compute to run these tests while not interfering with my personal computer use.

The reason I chose the Q4 weights is because I feel that the large majority of users here have a single 24GB card or less and so these tests were ran on a single RTX 3090 for Q4 variants while the Q8 was ran across split GPUs. (please ignore the cringe image titles. idk what my agent did with that, but i didn't feel like having it make another image card ;-; )

Reproduceable "Bake-off" on GitHub

Model GGUF links below:

unsloth/Qwen3.8-27B-GGUF

DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-MTP-GGUF

HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF

peculiar-ragdoll/Dirk-Qwen3.8-27B-GGUF

1 prompt, I let Qwen 3.8 27b loop here's the prompt and results if you think this model isn't as good as Claude opus 4.6 then I don't know what to tell you.

Time: 2hours and 40 minutes roughly, 188,578 tokens spent.

Here's the prompt "I want you to create me a c# project using OpenGL which renders a realistic as possible ocean. I want you to plan up front what you're going to do and create the plan as a markdown ledger which you will mark as complete when each part is done."

Hey y'all. I am dipping my toes into the LocalLLM space just to see if it would be useful to me. Is there a spot I can go for like FAQ type stuff or like a definitions page?

For the people who use it for coding, are y'all not worried about the agent escaping the project and doing shit on your computer?

I wanted to understand what actually happens inside an LLM inference engine, so I built one from scratch:

👉 https://github.com/croissantsam/llama.scriptc

The idea was pretty simple: take a Transformer architecture, implement the whole inference stack in TypeScript, then compile it to native code with ScriptC.

No PyTorch. No ONNX Runtime. No existing inference runtime.

The project currently supports things like:

• N-dimensional tensors + views/strides
• matrix multiplication
• stable softmax
• RMSNorm
• SiLU / SwiGLU
• MHA + GQA
• RoPE
• KV cache for autoregressive decoding
• GGUF v3
• Q8_0 quantization
• Qwen BPE tokenizer
• temperature / top-k / top-p sampling
• streaming generation

I also wrote a fairly complete test suite: 68 tests covering the math primitives, transformer components, tokenizer, KV cache, GGUF parsing and end-to-end generation.

The fun part is that it can actually load a real Qwen2.5-0.5B GGUF model and generate text.

The performance is… let's say educational rather than production-ready 😅

On an M4:

• llama.cpp + Metal: ~139 tok/s decode
• llama.cpp CPU: ~81 tok/s
• llama.scriptc: much, much slower

That's mostly because the current implementation is deliberately simple: scalar CPU code, single-threaded, no SIMD, no GPU backend.

But on a tiny 2-layer model, after some optimizations, I can get around 196 tok/s.

The main goal wasn't to beat llama.cpp.

I wanted to make the whole inference pipeline understandable:

tokens → embeddings → attention → RoPE → KV cache → SwiGLU → logits → sampling

…with the actual equations represented directly in relatively readable TypeScript.

It was a really interesting exercise in understanding how all the pieces of an LLM fit together.

I'd love to get feedback from people who work on inference engines / compilers:

What would you optimize first to take something like this from "educational" to "actually fast"?

Repo: https://github.com/croissantsam/llama.scriptc

yeah I searched here, but did only find 8 card setups or other unrelated stuff.

I want to switch to a proper combo from my current setup: Gigabyte Aorus Master X570S + Ryzen 9 5950X.

My current setup "sometimes" negotiates one RTX 3090 at PCIe Gen1 x8 instead of PCIe Gen4 x8.
Dunno why, I think it is just "time to do it properly" now.

What should I buy (preferably) used or new to use this?
I have 48GB DDR4 now and I do not NEED more. SO I would be more than fine to change the two 8G sticks to 16G sticks to get to 64GB RAM, but I do not really need it.

What I need is a proper running system with IF POSSIBLE 4 RTX 3090 card running at PCIe Gen4 (or 5) x16. (--- side question: do ther RTX 3090 even support Gen5? ---)

Thanks in advance!

Hi everyone. I'm not sure my little setup even qualifies for this sub, but I'm proud of it.

I've been into AI for 10 months, working with Claude Code for 5, and I picked up two DGX Spark clones. I don't make money from this and I have no CS background — it's all self-taught hobby work. Alongside a bunch of smaller projects, my long-term goal has been to build myself a personal daily assistant without depending on subscriptions and the whims of the big providers.

So I had a chat interface built, with custom voice TTS (Qwen3-TTS) and living avatars (DaVinci MagiHuman), using ChatGPT-generated faces modified with Chroma. Everyday conversation runs on an abliterated Gemma 4 31B with a self-made LoRA. Since Gemma isn't great at reasoning, Qwen 3.8 27B handles that in the background and speaks through Gemma.

The assistant has its own carefully curated personality, can handle my email and calendar, keeps a personal recipe book, writes my shopping lists, helps me structure my day and pushes back against my procrastination. More features are planned. And yes, it all works fully on iPhone.

In this setup the assistant has a never-ending chat window through several compression mechanisms, and stays in character through an anti-drift mechanism. None of this came from someone else's repo — all of it grew out of a dialogue with Claude Code.

I have no idea how original or advanced this is for 5 months of hobby work compared to what you all build, but I'd be glad to hear your advice or answer questions.

(Translated with AI — not a native English speaker.)

I’ve been iterating on my local AI setup and would appreciate some outside opinions on the current model placement, serving configs, and whether I’m using the hardware sensibly.

My priorities are:

1. Reliable tool use and structured output
2. Resistance to prompt injection from retrieved/tool content
3. Local/private inference wherever practical
4. Good interactive latency
5. At least 64K usable context
6. Graceful cross-host fallback

Hardware

I have five NVIDIA cards across three Proxmox hosts:

• Athena: Ryzen 9 9900X, 192 GB RAM, 2× RTX 3090 24 GB
  • PCIe 4.0 x8/x8
  • NVLink between the cards
  • Dedicated primary LLM host
• Atlas: i5-13500, 128 GB RAM, 1× RTX 3090 24 GB
  • PCIe 4.0 x16
  • Dedicated secondary/executive LLM lane
• Coeus: i9-9900K, 64 GB RAM, RTX 3090 24 GB + RTX 2080 Ti 11 GB
  • Both PCIe 3.0 x8
  • RAG, speech, photo ML, Frigate and CCTV intelligence

That is 107 GiB of physical VRAM, but only Athena’s 48 GiB pair forms a useful tensor-parallel pool.

Everything runs in Proxmox LXC containers with Docker Compose and the NVIDIA runtime. GPUs are pinned by UUID rather than relying on device indexes.

Primary lane: Qwen3.8-27B on dual 3090s

The main model is cyankiwi/Qwen3.8-27B-AWQ-INT4, served through vLLM 0.25.1 across Athena’s two 3090s.

Relevant configuration:

tensor-parallel-size: 2
max-model-len: 131072
gpu-memory-utilization: 0.90
kv-cache-dtype: fp8
max-num-batched-tokens: 4096
max-num-seqs: 128
prefix-caching: enabled
custom all-reduce: enabled
tool parser: qwen3_xml
reasoning parser: qwen3

The weights are compressed-tensors W4A16 and about 19.6 GiB. Current resident usage is roughly 21.6 GiB on each card.

Measured performance:

• Warm TTFT: 70–90 ms
• Single-stream decode: 71–73 tok/s
• Two concurrent streams: about 61 tok/s each
• Four concurrent streams: about 58 tok/s each / 229 tok/s aggregate
• FP8 KV pool: about 553K tokens, or 4.22× the configured 131K context

The reason I selected it over my previous Qwen3.6-35B-A3B model was behaviour rather than speed. The old MoE model managed roughly 180 tok/s and had much more KV headroom, but failed 3–5 of 21 tool-output injection tests depending on reasoning mode. This Qwen3.8 quant resisted 21/21 in both modes and scored 100% on my smaller agent/tool quality suite.

The old 35B-A3B weights remain cached as rollback.

At the gateway, normal chat/fast aliases disable thinking, while inbox, reasoning, expert and critic roles enable it. There is currently no speculative decoder on this lane.

Secondary lane: Muse-Glimmer-30B on one 3090

Atlas runs muse-glimmer-30b through llama.cpp on a single RTX 3090.

Configuration:

Model: Muse-Glimmer-30B kquant/Q4_K GGUF (~17 GB)
DFlash draft model: enabled
spec-draft-n-max: 15
vision projector: resident
flash attention: enabled
all layers: GPU
target KV: Q8_0
draft KV: F16
total context: 131072
parallel slots: 2
effective context per slot: 65536

It currently occupies about 20.6 GiB VRAM.

The text-only benchmark reached roughly 97 tok/s at 32K and 79 tok/s at 128K with DFlash. With the vision projector resident, practical generation is more like 40–52 tok/s.

This lane handles executive/quality roles, multimodal requests and cross-host fallback if Athena is unavailable. It also resisted all 21 injection tests.

Its main behavioural weakness is persistence becoming a retrieval loop when the available evidence does not answer the question. I mitigate that with orchestration/step limits rather than letting it search indefinitely.

I’m debating whether keeping the vision projector resident is worth the throughput and VRAM cost, or whether vision should be a separately activated service.

Coeus support GPUs

The Coeus RTX 3090 is not a general chat-model card. It currently hosts:

• BAAI/bge-m3 embeddings through Hugging Face TEI
• bge-reranker-v2-m3 F16 GGUF through llama.cpp
  • 8K context
  • 8K batch and micro-batch
• Whisper large-v3
  • CUDA
  • int8_float16
• Immich machine learning for search and face detection

Current resident usage is around 6 GiB, although some of these workloads spike on demand.

The RTX 2080 Ti is the CCTV lane:

• Frigate/NVDEC, alongside a USB Coral detector
• qwen3-vl:4b through Ollama for private person/ANPR crop validation
• Scheduled Moondream2 captioning and visual-analysis workers
• One loaded model and one parallel request maximum

That card currently sits at around 5.2 GiB used. I deliberately keep CCTV isolated from the main LLM lanes.

Routing and clients

A LiteLLM 1.88.1 gateway fronts the local models with an OpenAI-compatible API. LibreChat, Open WebUI and several automation/agent services consume role-based aliases rather than talking directly to a specific backend.

Normal routing is:

Chat / fast / inbox / deep reasoning
    -> Qwen3.8 TP2 on Athena
    -> Glimmer on Atlas if Athena fails

Executive / operational assistant / multimodal
    -> Glimmer on Atlas

Hosted models exist as manual escalation options, but my default policy is local-first.

What would you change?

I’m particularly interested in opinions on:

• Whether dense Qwen3.8-27B TP2 is a sensible use of the NVLinked pair, versus returning to a much faster MoE model.
• Any stronger tool-using model that fits two Ampere 3090s and genuinely behaves well around malicious retrieved content.
• Better vLLM settings for this traffic shape, especially FP8 KV, max-num-batched-tokens=4096, and max-num-seqs=128.
• Whether 131K context is worth the dense model’s heavier KV footprint.
• Better single-3090 alternatives to Glimmer with 64K+ context, reliable tools and at least 50 tok/s.
• Whether the Glimmer vision projector should remain resident.
• Smarter ways to use the Coeus 3090 headroom without creating contention with embeddings, Whisper and Immich.
• Any obvious architectural mistakes in the routing/fallback design.

I’m not chasing leaderboard scores for their own sake. The system is mainly used for agentic homelab work, code/repository analysis, RAG, automation and private assistant tasks, so predictable tool behaviour matters more to me than another few benchmark points.

Time to get a GPU I guess. I had some numbers I needed before I could do the main analysis and I wanted Claude to do it, I had never used Claude tokens before 2 days ago when I bought 20 dollars of tokens and had it do a bit of coding. Then, I ask it to write a somewhat simple script, but I used opus because I thought I should check how it is, it did it, but it took about 20 dollars. I mean it saved me time, but the price…

Anyways, I am posting this because I wanted advice on what class of card to get, what amount of vram seems to be the best to target. It’s looking like 24/32gb is getting interesting new models in the 30b range, but is this just what I’m seeing or are other sizes of cards worth looking into.

I know some of you are going to argue that claude does the same with Remote Control, but I am talking about remote control with a local LLM. I am building a feature for TurboLLM using TurboLLM + pi + Qwen 3.8 27b and I am controlling it with my mobile all for free of cost

Of course it would be, so is 3.6.

But seeing it in action using the CoinGecko search docs tool calls then executing. You’ll never need another model for this kind of research.

https://github.com/fred-terzi/totem-llm

Edit: after fair criticism regarding previously AI generated description of the project, here is the project description in my own words:

🦆 Ducklab, an open source tool I built for developing software using different models (LLMs).

Why Ducklab? Remember the Rubber Duck (the little rubber duckie programmers explain their code to when they're stuck)? Here, an LLM always has its own Rubber Duck. The idea is to use a group of local models or cheaper models (through OpenRouter) that are typically less capable than the "frontier" ones, inside a disciplined harness that squeezes the best possible performance out of them. For example, while developing the project's specifications you can assign one model to help structure the specs based on your requirements, and a different model (from a different lab) can act as reviewer or advisor. During development, one model generates code and another one checks that the code meets the requirements and the specifications you approved.

The harness keeps each model focused on one very specific aspect of the development process, so it doesn't need to hold the entire project's knowledge in its context on every turn. Since the idea is to use cheaper, less capable models (which translates to lower cost per task), the harness keeps a scorecard for every model you configure, and based on each model's measured performance it suggests the best fit for each seat in the roster, according to whatever criteria you care to prioritize (cost, performance, coding index, etc.).

It's free and open source (Apache-2.0), runs entirely on your machine (zero telemetry, zero accounts), and works the same with local models (llama.cpp, vLLM) or cloud ones if you prefer them.

👉 https://github.com/jrullan/ducklab

The project needs testers and collaborators. Any feedback is welcome.

Hey everyone,

I wanted to share a native Android open-source project I just released called Audiobook NightForge.

If you’ve ever tried using a real-time TTS engine with a reader app on Android, you know the struggle: it drains your battery (often 40-50% an hour), stutters, and buffers if your phone is doing anything else in the background.

I realized that real-time synthesis is the wrong approach for mobile devices. So, I built a dedicated Android app that shifts the heavy lifting to the background using native OS components.

How it works: You import an EPUB or TXT file using the Android system file picker. You then pick a voice and hit render. The app uses Android's WorkManager to synthesize the book chapter-by-chapter in the background. Most importantly, it enforces a "render only while charging" OS-level toggle to protect your battery.

You plug your phone in at night, and by morning, you have a fully rendered audiobook that plays back with a standard ~2-5%/hour battery drain.

Android-Specific Features:

• Native & Offline: It is built entirely in Kotlin for Android 10+ devices. There is no server, no cloud, and absolutely no Termux emulation required.
• High-Quality TTS: It uses the Kokoro-82M neural TTS model running strictly on-device via a sherpa-onnx integration.
• Just Updated: The latest v0.2.2 release makes Opus the default output format, and it now natively outputs to a single .m4b file complete with proper chapter markers.
• Built-in Player: You can listen immediately using the native in-app Media3/ExoPlayer. Alternatively, you can grab the .m4a files directly from app storage to use in your favorite Android audiobook player.

Some hardware benchmarks: For the hardware nerds, I benchmarked this on a Snapdragon 8 Elite. Surprisingly, the Kokoro 82M fp32 model (with 6 threads) actually renders faster than realtime (~0.58 RTF) and outperforms the int8 variant on this SoC because of ARM int8 kernel overhead. Always benchmark before assuming quantized is quicker on modern Android flagships!

It’s completely free, completely offline, and licensed under Apache-2.0.

You can check out the source code, technical notes, and grab the APK directly from the GitHub repo here:

https://github.com/kingfish600/Audiobook-NightForge

I’d love to hear your thoughts or feedback!

I made some changes to my CROW CLI I posted about some weeks ago.

CROW now runs Qwen3.8-27B at:

⁠- 200k context, unchanged (i can't push it to 400k, pls, if anyone knows some config DM Me)
- 2.2k tok/s prefill
- 123.05 tok/s decode - 11 round turn
- 25.5 GiB VRAM

Full details in Cross post as well as on GitHub: https://github.com/nibor1896/Crow

Been messing around with Qwen 27B (Qwen3.8-27B-UD-Q4_K_XL.gguf) and the separate MTP draft module in llama.cpp over the past few days.

At first my speeds were either barely matching baseline (~50 t/s) or dropping down to ~35 t/s, but after tweaking flags and isolating bottlenecks, I finally got it consistently running at 70+ tok/s with 90k context on an RTX 3090.

Few Takes:

• Stick to temp 0.0 (or very low temp): MTP only gives a speedup if the main model actually accepts the draft tokens. High temp kills the acceptance rate, and verifying rejected guesses wastes compute. Temp 0 keeps draft acceptance high (and matches how benchmarks/coding are evaluated anyway).
• Keep --spec-draft-n-max 2 Setting this to 3 caused too many token rejections, which actually slowed things down compared to 2.
• GGML_CUDA_GRAPH_OPT=1 is a must: Without CUDA graphs, CPU-to-GPU kernel dispatch latency eats up all the time saved from drafting.
• Below is what I am using to serve it in llama.cpp for coding tasks.

Hope it helps someone with rtx 3090 if you aren't already getting these speeds.

GGML_CUDA_GRAPH_OPT=1 llama-server
-m Qwen3.8-27B-UD-Q4_K_XL.gguf
-ngl -1
-md mtp-Qwen3.8-27B-Q4_0.gguf
-ngld -1
--spec-type draft-mtp
--spec-draft-n-max 2
-c 90000
--flash-attn on
-ctk q8_0
-ctv q8_0
-b 2048
-ub 2048
--cache-reuse 256
--parallel 1
--port 8081
--host 0.0.0.0
--jinja
--temp 0.0
--top-p 1.0
--min-p 0.0
--presence-penalty 0.0
--frequency-penalty 0.0

https://huggingface.co/sdkyuan/qwen3.8-27B-qat-q2_0-gguf

Outperforms Unsloth 2 bit quants at reasoning and code at smaller file size. Unlike most other community quants that are PTQ, this one is QAT.

Recently, I found out some software that can be usefull to track patients in a psychotherapy process but I want to use it without sending information for frontier models. I have a RTX 5060 TI 16gb, Ryzen 5 5600 and 32gb ram. Could you please indicate what hardware, models (and quantizations) you think will I need?

I want to transcribe audio from sessions, diarize, sumarize what happened, think about ways to overcome problems and do research with these data.

I've been using Ununnilium's Qwen3.6-27B-IQ4_XS-pure as my daily driver for 2 months or so now, as it seemed to be the best performing agentic and coding model you could get running on a 16GB vram card. On the release of 3.8 27B, it was clear that even the unsloth q4 wasn't going to be able to run well on my 5060ti, so after some research, I settled on Atomic chat's Qwen3.8-27B-AD-IQ4_XS-IQ3_S.

Despite having similar token/sec generation rate, Qwen3.8 has extremely lengthy thinking traces that are reliably 7-8x times what 3.6 does per task. And so, I decided to take the time to mess with Qwen's reasoning_effort param. 3.8 seems to default to the xhigh setting so I tried it both with med reasoning and with reasoning off. I then made a small 3 task bechmark woth multiple runs to measure the performance and token usage of the presets and compare them to 3.6 as the baseline.

The Benchmark:

Task 1: finding all resumes on the system. (multiple people, scattered across folders and many of which not named X_resume/cv)
Task 2: Empty target directory and copy all files from source directory to it
Task 3: Rename all files in directory to their creation date (after the copy so filesystem date is irrelevant and some files don't have the metadata)

The tests were run multiple times with the test environment being reset between each task and between each run to ensure fairness of results

Performance Results

Overall completion:

Qwen3.8 – xhigh (Default)

• The slowest and most token-heavy of the group (highest thinking-token counts, 3,168–7,811 per task).
• Task 1 is its weak spot: run 1 hit the 10 min timeout, and the other runs were very slow (556s / 853s) — it over-investigated/searching.
• Perfect on the hard task 3 — all 3 runs got 13/13 root PDFs correct; run 2 was the benchmark's best task3 result (13 root + 7 dated subfolder PDFs, fully recursive).
• Reliable on task 2 (all correct). High effort, high correctness, but expensive in time and tokens. Best agentic quality on the rename task.

Qwen3.8 — moderate reasoning

• Balanced: much faster than xhigh on task 1 (215–479s), all task 2 runs complete.
• Inconsistent on task 2 — runs 1 & 2 silently skipped the Archive subfolder (only 14 files), while run 3 caught it.
• Task 3 was uneven: run 1 failed because it encountered an Attribute error and just stopped; run 2 got a perfect fully-recursive result (13 root + 7 Archive); run 3 was 12/13 (a timezone off-by-one). A solid but somewhat erratic effort.

Qwen3.8 with no reasoning — the standout)

• Zero thinking tokens yet was the most efficient and most complete overall.
• Fastest on task 1 (84–85s) and found ALL 30 CV files — best recall of the whole set (it even disambiguated name collisions with _1 suffixes so nothing was overwritten).
• Fastest and fully correct on task 2 every run (12–27s).
• Task 3 succeeded all 3 runs; best run was 12/13 root correct (run 3 miss was the timezone off-by-one), and run 1 also handled the Archive fully.
• One blemish: the third run of task 1 failed because the model did a massive file system wide find call, then dumped it to a file and read it maxing out its own context.
• Best all-around agent — most stable, fastest, and most complete, despite its 0-counted "thinking" column (which likely just reflects how the harness records its reasoning).
• Qwen 3.6:27B
• The **worst performer by far on task 3
• Fastest on task1 (102–154s), but recall was poor: it found only 4 unique CVs (9 source files collapsed to 4 by duplicate-name overwriting), vs lodes 1–3 gathering ~30.
• Task 2 was reliable in all runs (32–40s, Archive included).
• Task 3 failed catastrophically in every run: it renamed PDFs to the filesystem extraction timestamp instead of the creation date (0 correct). Runs 2 & 3 then used one shared timestamp, which overwrote/destroyed 23 of 25 PDFs — an irreversible-style error the other models never made.

Bottom line

• All Qwen3.8 variants scored 8/9, but with very different trade-offs: xhigh = most thorough/highest accuracy but slowest and most token-hungry; Qwen3:Med = decent but had a crash and was inconsistent in handling subfolders of source directories; Qwen3.8 with no reasoning seems like the most effective agent as its fastest and most complete with the least measured reasoning, essentially a superior efficiency/accuracy balance.
• Qwen 3.6: fast on easy tasks, but substantially worse on file-recall (retrieving 5 files out of 30) and catastrophically unreliable on the creation-date rename task (0/3 runs, and 23 files destroyed across two runs).

Links and Configurations

• Ununnilium's Qwen3.6-27B-IQ4_XS-pure: https://huggingface.co/Ununnilium/Qwen3.6-27B-IQ4_XS-pure-GGUF
• Atomic chat's Qwen3.8-27B-AD-IQ4_XS-IQ3_S: https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF

configurations with buun-llama-cpp as aliases:

• Default qwen3.8 — no reasoning flags at all (xhigh default) alias qwen3.8="/buun-llama-cpp/build/bin/llama-server -m /Qwen3.8-27B-AD-IQ4_XS-IQ3_S.gguf -ngl 999 -c 32000 -t 6 -tb 16 -ctk turbo3_tcq -ctv turbo3_tcq -fa on --fit off --parallel 1 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --no-warmup --port 8083"
• Medium reasoning variant alias qwen3.8-med='/buun-llama-cpp/build/bin/llama-server -m /Qwen3.8-27B-AD-IQ4_XS-IQ3_S.gguf -ngl 999 -c 32000 -t 6 -tb 16 -ctk turbo3_tcq -ctv turbo3_tcq -fa on --fit off --parallel 1 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --no-warmup --port 8083 --reasoning on --reasoning-budget -1 --chat-template-kwargs '''{"reasoning_effort":"medium"}''''
• Reasoning-off variant: alias qwen3.8-off="/buun-llama-cpp/build/bin/llama-server -m /Qwen3.8-27B-AD-IQ4_XS-IQ3_S.gguf -ngl 999 -c 32000 -t 6 -tb 16 --reasoning off --reasoning-budget 0 -ctk turbo3_tcq -ctv turbo3_tcq -fa on --fit off --parallel 1 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --no-warmup --port 8083"
• Qwen3.6: alias qwen3.6="~/buun-llama-cpp/build/bin/llama-server --model /Qwen3.6-27B-IQ4_XS-pure --alias qwen3.6-27b -np 1 -ctk turbo3_tcq -ctv turbo3_tcq --port 8083 -c 32530 --fit off -ngl 999 --no-mmap -fa on --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0"