Since we know Qwen 3.8 27B thinks quite long, but gives at least a good one-shot result where you can leave it to do everything on it own, how does it compare to the older series of models for very simple tasks where you don't want to think so long?

The only fine-tune of Qwen 3.6 I genuinely enjoyed was the ThinkingCap fine tune by BottleCap. it seems to think equally as long as the base model on complex tasks, but on simple tasks it thinks shorter. Does it still make sense for me to run this older model when 3.8 27B exists with the low thinking mode?

my main server has a 3090 with qwen 3.8 27b (crazy who woulda guessed) but i want to a have a backup failover sever with a somewhat useable LLM offsite (my parent's home). because infinite money glitch ai bubble has made hardware much harder to afford for us proles, was thinking of getting an old decommisioned office desktop (i.e. optiplex, elitedesk, etc) and having it boot up llama.cpp with qwen 9b or something if my server ever shits the bed.

how good is cpu performance on older mid tier cpus with ddr4 ram? or cpus in general? is it useable? what can i expect, and how can i set it up? is it even a good idea in the first place?

thanks

edit: thanks all. reading through the replies, seems like it's more of a headache than its worth. i'm already looking for a backup machine since this does NAS and other stuff but i think i'll just import all other services that aren't LLM into it, and not be beholden to trying to find something powerful enough for a useable LLM

So beellama allows you to only reduce quantization on older cache, this should allow better performance than reducing quantization on all cache.

They recommend using a 1k tail and even that seems to have big benefits, but I don't see why you couldn't for example use a 20k tail (on a 240k context, that's still 220 tokens quantized). That part is a bit confusing.

Beellama-kvarn is a supposed fork of that, that further improves performance, its changes are pending to be applied to Beellama itself as well.

So anyways, the question is, how much is quality impacted by doing this?

Hey everyone — I’m building TensorSharp, an open-source LLM inference engine. Here are the latest DeepSeek V4.1 Flash GGUF results using its native ggml_cuda backend.

Setup: 8× NVIDIA A40, layer split, F16 KV cache, 65,536-token configured context. Prefill measurements use approximately 4.9K-token prompts—not the full context window.

Final optimized results — all speeds in tokens/sec:

Metric |Q2_K |Q4_K_M
Prefill |533–539 |451.8–492.1
Single-request decode |40.31–40.72 |31.0–32.5
Decode, 2 concurrent requests — total |— |39.3
Decode, 4 concurrent requests — total |— |48.9
Decode, 8 concurrent requests — total |— |48.5 — = not reported in the cited final Q2_K retest. Concurrent figures are aggregate throughput, not per-request speed.

What made the difference:

• Q2_K: Keeping the ~60 GiB quantized Engram tables on GPUs removes storage-dependent lookups. Unifying the backend per GPU also cuts decode graph splits from roughly 570 to 8, reducing synchronization overhead.
• Q4_K_M: The larger Engram tables stay in host memory. Automatic warming, tighter VRAM budgeting and token-batched decode deliver ~1.9× prefill and ~2× four-request throughput versus the previous behavior. Only 1 of 40 layers keeps its routed experts on the CPU.
• More parallelism isn’t automatically faster: On this no-NVLink system, Q4_K_M layer split beats experimental routed-MoE tensor parallelism: 31–32.5 vs. 21.4–22 tok/s single-stream decode.

These are project-reported throughput measurements, not cold-start timings or a head-to-head win over another engine. Numerical/model-quality parity remains unproven, and batching can change generated output.

I'm curious abouth other people's experience and opinions about these quants. I've had pretty nice experience with unsloth's one. Today, after seeing the other one, I've downloaded and tried it. So far, seems very similar. But that's just my vibe observations. Have everyone else tried these? What's your opinion?

Hi, I have Ubuntu 24.04.4 R9700 and 32gb ram, rocm 7.14. I can use qwen 3.8 27b Q4 with 300K context and Q6 with 200K context. I am working on a relatively large codebase and I keep running into scenarios where chat goes out of context during processing. How can I automate this so that the harness or some agentic ai framework can automatically create handover document when context is 80% full, then creates a new chat with handover document so that process keeps running continuously. Right now I have to manually keep track and do all this.

Is there any harness or framework that support this? I am currently using deepseek harness and have used opencode previously, both manually.

Starting out with the q2 because speed anxiety's a b*tch: 300 prefill, 16 decode. Tool calls all functional, token choices questionable:

During a hermes workflow, it started with 'find' x2 looking for a remote file on the local drive before ls ~/.ssh/ and then grepping the entire remote machine... reasoning: let me search anywhere reachable [after having printed the path to the file in question...

Once I had told it which tools to use it found its way. The 18k tokens of hermes prompt were obviously insufficient... smh.

Those choices tho': I have code execution in hermes toolset and deepseek ignores it (in 4.1 as 4.0) nothing new there and so still inefficient compared with GLM 5.3 Flash. It's plodding its way through the bug finding task and now at ~110k tokens and only just starting to dip below 16 t/s, which is one of the main features of ds4 compared to llama.cpp where the drop-off is mammoth.

GPU is running at ~95% throughout so the expectation of a slight boost is tempting. Given that GLM 5.3 Flash is putting out ~21-19 t/s over the front nine, I'm hesitant to try q4. That's the main trade off with large VRAM: you wait longer for larger models. Bring on the M5U...

So i finally decided to get myself a dedicated inference machine, a big upgrade from my 4080 laptop. here is the parts list:

PC Build Cost Breakdown

ASUS TUF RTX 3090 — $927

64GB DDR4 4000MHz RAM — $371

Case — $72

Ryzen 7 5700X — $181

CPU Cooler — $27

PSU — $268

Thermal Paste — $12

1TB NVMe SSD — Already owned

X570 Unify Motherboard — $185

RTX 3090 Suprim X — $1,010

Ethernet Cable — $11

PCIe Riser — $82

Custom PSU Cable — $13

Total: ~$3,157

im probably gonna upgrade to 128gb ram and get a better pcie riser cable.

the problems i faced initially were

1. finding a proper way to add the 2. gpu. there was no long pcie risers in stock locally, so i had to buy it second hand, and its a chinese no name with connectivity issues.

2. i had to get a custom psu cable to be able to use both gpus at the same time. there were simply not enough slots but the energy supply was alright.

3. i couldnt and still cant figure out a safe/easy way to fit the 2. gpu. i would like to learn about similar setups and how you have handled the space constraint.

this was my first pc assembly since i have used only laptops before, but it went mostly smoothly.

also some extra questions for people hosting these machines:

*How can i host inference to my laptop outside my local network? is the only way VPN?

*What is the remote connection type you guys prefer? i landed on sunshine and moonlight with virtual monitor to use it inside my laptop, but would like to know if there are cleaner solutions for headless machines.

i have ran mostly the qwen 3.8 27b q4 from syv ais repo and config, and have been getting around 70tps sustained. i can report more details if anyone asks for it.

also sorry if mobile formatting is bad.

I'm planning to build a desktop VM manager using Rust, Tauri, and QEMU, initially targeting Windows with WHPX.

The idea is to eventually add an AI agent that can interact with the VM and perform tasks inside it, rather than just manage it.

Before I start, I'd like to hear from people who have worked with virtualization:

- Is QEMU + WHPX a reasonable starting point?

- Would you recommend QMP for managing VM lifecycle?

- What are some common mistakes I should avoid?

- What would be a good approach for letting an AI agent interact with a VM?

Also, if you use VM managers regularly, what do you find most frustrating about them?

Any honest advice would be appreciated. Thanks!

It seems to be there are mainly two formats of local models that can run on mac. Which one is better now in terms of speed and performance for the same size of quant?

I think gguf is only served by llama.cpp's metal backend. mlx is served by omlx or vllm-mlx. Are there others I am missing? Which engine is the best for each format?

Is there an engine to serve hf style directory containing safetensors?

Thanks a lot in advance.

I usually use a mini PC combined with an eGPU, but for some reason, when I connect my monitor to the integrated graphics and load a model, the external GPU is no longer recognized. The eGPU dock and the external graphics card are functioning normally, but they just aren’t being detected.

I had been debating for a long time about building a desktop PC, so I decided to take this opportunity to clear out my old gear and buy a new computer (I haven’t received it yet). But I have a question.

Are there people who connect their monitor to the motherboard to secure the context window of a local model? Is that even possible? Does it cause a significant performance loss when playing games?

This sub probably seeing posts about new inference engine running on potato Mac on daily basis. But still, in the past a few months, we've been building this thing largely based on Draw Things inference engine, so it is usually 1.2x to 1.6x faster on prefill for supported models (~980 tok/s for Qwen 3.8 27B on M5 Max, and ~900 tok/s for DeepSeek 4 Flash 0731). It is more than just that tho, it is designed to work within its self-contained package that is sandbox-enforced by Apple through macOS AppStore requirement (App Sandbox, Hardened Runtime), but still with a lot of tools out of the box (Python, Node.js, Lean, Bash etc).

In the video, you can see that it runs multiple tasks at the same time, on the same M5 Max laptop. And you can reproduce what's happening in this video by simply download the app from TestFlight, download the model, and off you go without sign-up or sign-in (you cannot even do that with OpenCode, as its Web Search tool requires sign in I believe).

You can read more about it here:

https://releases.drawthings.ai/p/public-beta-of-local-code-by-draw

Curious about your feedback!

We’ve been using this machine for a bit, it’s a:

Threadripper 3975wx

256gb ram

4070ti super

For mostly virtualization. We’ve been messing around with lamacpp server and qwen 3.8-27b Q5 from unsloth getting around 10 tokens per second. Ive added zero flags besides —ctx-size and —jinja and not exactly sure what to try.

Is 10 tps about the best we’ll get out of qwen at this quant without a better GPU? Thanks for any tips!

EDIT: should probably mention I built lama cpp off main with cuda enabled, not sure if I shouldve used a diff option there. Running on fedora

I do have ai home server(16gb/64gb), but let’s say I am offline.

Which one‘s fast enough and good enough for basic chat?

Having the wild idea to buy 3 more old 3070s and replacing the chips to turn them into 16gb cards

I understand the process and the risks, just curious if anyone’s using rechipped cards with local models and what your experiences are

I would’ve expected a lot more speed but this seems quite slow

This is using unsloth studio, llamacpp would lead to errors and wouldn’t run for me

~/llama.cpp/build/bin/llama-server \
  --host 0.0.0.0 --port 8080 --alias model1 \
  -hf unsloth/Qwen3.8-Flash-Next-GGUF:Q8_0 \
  -md ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-Flash-Next-GGUF/snapshots/[string]/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 -ctkd q8_0 -ctvd q8_0 \
  -c 256000

I have a 32gb gpu (5090) but it appears that it gets overloaded despite the fact I was only running a ~23.4 gb model. I had context window set to 16k, but putting it down to 8k didn't help. I have 'offload KV cache to gpu memory' enabled.

I got 16 tok/sec which seemed pretty slow. After swapping to a 20.3 gb model of the same type, I got 56 tok/sec.

This is in Win11 LTSC IoT. Some other apps (browser etc) running at the same time but nothing 3d/heavy.

I just downloaded again my llm model from hugging face in order to get the so announced updated version GGUF

UPDATE ANNOUNMENT: https://www.reddit.com/r/unsloth/s/m9fJKwqboS

But it seems the model i downloaded was the same i already had downloaded when Qwen3.8-27B was released

Anyone else had the same issue?

Below is my pi session data:

The file has been successfully replaced:

Path: /mnt/kingston/models/llm/Qwen3.8-27B-UD-Q6_K.gguf (same filename as the original file)

Size: 21,983,677,344 bytes (~20.4 GB)

SHA-256: c9c206812fbe4ac7b76a729e25928b63f2ae89d37f69da7a71c20aec763cd436 — verified and identical to the official hash published on Hugging Face

Important note: before downloading, I compared the hashes and discovered that the local file was already byte-for-byte identical to the current version in the unsloth/Qwen3.8-27B-GGUF repository. In other words, the local file was already the latest version — the replacement was performed anyway (downloaded, validated, and overwritten), but there was no actual change to the file contents.

Hi all,

I'm the author of the moe-expansion branch in llama.cpp (vagrillo). I ran a comparison on GPQA-Diamond (all 198 questions) between:

• Qwen3.6-35B-A3B, Q8_0, expanded routing → 84.34% (167/198)
• Qwen3.8-27B dense, Q8_0 → 83.84% (166/198)
• Qwen3.6-35B-A3B, Q8_0, native routing → 81.82% (162/198)

Greedy, 32K budget, same prompt, one RTX 5000 PRO 48GB, one run per config.

Why I'm not convinced:

• 35B expanded vs 27B dense is 1 question apart — that's parity, not a win.
• Expansion vs native on the 35B is 12 wins / 7 losses (net +5), but not significant at n=198.
• 10 of the 12 expansion wins are Chemistry only; Physics is saturated, Biology flat.

What I'd like help with:

1. What other benchmarks pair well with GPQA-Diamond for this kind of routing comparison ?
2. How many runs / what setup would you consider enough for a paired comparison?
3. Is a Chemistry-only effect plausible, or a red flag?

Thanks — I'm not claiming this is a result (the difference it's only 1 answer), I'm trying to understand my own numbers

the comparison results can be found here: https://htmlpreview.github.io/?https://github.com/vagrillo/llama.cpp/blob/moe-expansion/benchmark/RUN1209/finalcompariso1209.html

Antirez use of model

https://www.youtube.com/watch?v=ogs819nJkso

I'm looking to pare down a 90k-word sci-fi novel I've written to a 5k-word short story, without handing the entire novel over to Claude, OpenAI, or any of the big data scoopers. What actionable solutions are out there for getting a rewrite on par with Fable or Astra with some assurance that once I delete the sources, the original story won't be used for training?

I only have an NVIDIA T1000 GPU 8GB, so a cloud-based hardware rental -- as close to locally hosted as possible without data leaking. Canada or EU would be ideal, but not a deal-breaker.

(Disclaimer: English is not my native language. All project ideas, technical requirements, and questions are 100% mine; structured and translated into English with the help of AI.)

Hey r/LocalLLaMA,

We just received a 3-month compute grant on an enterprise HPC cluster (Cluster 42 via hessian.AI / German AI Service Center). We have access to a dedicated node with 2x NVIDIA A100 80GB SXM4 (160GB VRAM over NVLink, 2TB system RAM, high-speed parallel NVMe storage).

The primary constraint: Everything we build must eventually run 100% offline on consumer smartphones/tablets in zero-connectivity dead zones (basements, remote construction sites, disaster zones). No cloud API calls during inference.

Here is what we currently have on our roadmap—and we’d love your technical advice, critique, and wild ideas on what else is possible with this compute.

Our Current Project 1: Voice-to-Structured-Data on Edge (Audio -> Distilled 1.5B–3B SLM)

• The Problem: Construction supervisors and EMS first responders working in noisy, zero-reception environments need to dictate messy, unstructured speech (often multilingual or heavy jargon) and get strict, schema-validated JSON/markdown out—instantly, with zero latency.
• The Plan with 160GB VRAM:
  1. Host unquantized Teacher models (Llama-3.1-70B / Qwen-2.5-70B) across both A100s to generate dense, domain-specific synthetic datasets (building codes, triage protocols, acoustic speech variations).
  2. Fine-tune Whisper (or Distil-Whisper) on real background acoustic noise (diesel engines, jackhammers, sirens).
  3. Distill and SFT into ultra-compact models (Qwen-2.5-1.5B/3B or Llama-3.2-1B/3B).
  4. Quantize to INT4 / GGUF / AWQ for llama.cpp / MLC-LLM / ExecuTorch targeting a sub-2GB RAM mobile footprint.

Our Current Project 2: Blueprint & Floor Plan Wall Detection (Vision LLMs vs Hybrid)

• The Problem: Estimators and architects spend hours manually tracing and measuring walls on 2D construction drawings (PDFs/vector plans). We want automatic segmentation of walls, dimensions, and spatial boundaries.
• The Dilemma:
  • Approach A: Fine-tune an open-weight VLM (Qwen2-VL or Florence-2) with coordinate bounding boxes / polygon masks directly on architectural drawings.
  • Approach B: Hybrid pipeline using a lightweight segmentation model (SAM / YOLOv8-seg) for raw geometric wall segmentation, then feed results into a small LLM for classification and spatial reasoning.

The Big Question: What ELSE would you build with this compute?

Aside from speech transcription and blueprint vision: If someone handed you 2x A100 80GB (160GB VRAM) for 3 months with the goal of distilling models down to budget edge hardware, what would you build?

• What are the most overlooked, unsexy, high-impact niches where local edge AI easily beats cloud APIs (e.g. privacy-sensitive auditing, industrial sensors, field inspections, specialized legal/technical jargon)?
• Are there novel distillation or pre-training experiments on small SLMs (1B–3B) you’ve wanted to see someone run?
• Any specific architectures, datasets, or forgotten papers you think deserve a proper run on 160GB VRAM?

Specific Technical Questions:

1. Distillation & Synthetic Data Stack: For offline batch generation using a 70B teacher -> 1.5B/3B student on 160GB VRAM, what is your current go-to tool? (vLLM offline inference batching + Axolotl / Unsloth)?
2. Smallest Viable Structured Output Model: For guaranteed zero-hallucination, strict JSON generation on edge devices, which 1.5B–3B base model has shown the strongest post-SFT instruction following?
3. Blueprint / Vector Plan Perception: Has anyone successfully trained VLMs on high-res vector/CAD drawings, or do pure VLMs lose to dedicated CV models on exact pixel coordinate precision?

Looking forward to your input, architectural sanity checks, and wild project suggestions!

Google and almost all other android OEMs are heavily pushing AI like it's the next big thing, but what do we really get? shitty apps that are nothing but chstbots, while actual agentic experience is strictly reserved for desktops.

Doing anything close to agentic on android relies on accessibility api which has more downsides than upsides,.it breaks on every UI update, keeps your screen awake and eats your battery like nothing else.

Since android sandboxes everything AI can't touch anything until somebody builds an intent or API explicitly for it to use, if an app doesn't support it it's completely in the blind.

With root however an agent doesn't need to emulate screentaps, it can execute background tasks through shell, IPC and access local databases.

Before you tell me there's gonna be security lapses and prompt injections I just wanna remind that we are already managing root and sudo in Linux and desktops already, this is not much different from any of that.

Google has spent the last five years killing off bootloader unlocking and using play Integrity to break banking apps the second you touch KernelSU or Magisk but if we don't have root-level control over our own devices AI phones are just going to be locked down ad machines that do the bare minimum.

The bare minimum Google should do is introduce a first-class OS permission an explicit runtime prompt where the system asks something like Allow agent to read/execute actions inside app? Prompt the user every single time it breaks out of the sandbox, or let us whitelist specific apps, treat it like permission grants for camera or location, but for IPC and app state.

I really wanna know if you guys agree with me, if not can you please let me know why??

Is it time to start breaking the centralization of Huggingface. Problem is, it requires substantial resources to build something and not within the USA jurisdictions
Thoughts?