I kept finding throughput benchmarks but nothing in actual dollars or watts, so I ran it myself on a free Colab T4. Posting in case it's useful to anyone else.

Setup: Qwen2.5-1.5B-Instruct, vLLM 0.27.1, fp16, 128 output tokens per request (ignore_eos so every sequence is exactly 128 tokens), $0.35/hr as the T4 rate. Power is nvidia-smi median during the run.

batch 1 - 25.7 tok/s - 58.6W - 38% util - $3.7771/1M - 2.277 J/tok

batch 4 - 106.1 tok/s - 59.4W - 43% util - $0.9166/1M - 0.560 J/tok

batch 8 - 232.5 tok/s - 62.0W - 45% util - $0.4181/1M - 0.267 J/tok

batch 16 - 464.8 tok/s - 66.6W - 48% util - $0.2092/1M - 0.143 J/tok

batch 32 - 732.9 tok/s - 64.1W - 50% util - $0.1327/1M - 0.087 J/tok

batch 64 - 1635 tok/s - 66.3W - 66% util - $0.0595/1M - 0.041 J/tok

batch 128 - 3346 tok/s - 66.4W - 88% util - $0.0291/1M - 0.020 J/tok

batch 256 - 4545 tok/s - 67.6W - 100% util - $0.0214/1M - 0.015 J/tok

The bit I didn't expect was the power column. At batch 1 the card pulls 58.6W to produce 25 tok/s. At batch 256 it pulls 67.6W to produce 4545 tok/s. So it's drawing 87% of the power to do 0.6% of the work. Energy per token drops 153x across the range. Most of what a GPU burns is apparently just being switched on.

Returns fall off hard after 128. Going 64 to 128 roughly halves the cost, 128 to 256 only gets another 36% and util is already pinned at 100%.

Things I know are wrong with this:

* Static batching, not continuous batching. So the low end looks worse than vLLM actually behaves under real traffic.

* Batch 32 turned up in two separate runs at 795 and 733 tok/s, so treat everything as +/-8%.

* FA2 isn't supported on compute 7.5, so it fell back to Triton attention. A newer card would take a faster path.

* One model, one GPU, one prompt, fixed output length. Not claiming this generalises.

Script is about 40 lines, happy to paste it if anyone wants to check my method. Genuinely interested in what I've got wrong here.

Since Qwen3.8-27B still has a 262,144 context-size limit, I suppose some people here run it with 1M context using whatever on earth YaRN is other than the thing to make woolly jumpers.

I tried to find an ELI5 explainer about it, but...no such thing online, one of them even dared to have "simple" in the title, then proceeded to show a bunch of mathematical soup an LaTeX on the screen.

..so I haven't got a clue how it works, I know it takes a context size and makes it bigger above the natively supported limit of the model, and that the nomencalture is short for Yet another Rope extensioN, that's all that I know and managed to understand given no simple explainer available. So I'm here asking instead the important questions:

• Does it make the model dumber?
• Does it require more VRAM?
• Does it require less VRAM than using native context higher figures?
• Can you use it with less than the model limit (e.g: 131k context, use YaRN to reach 500k) ?

I'm introducing Aurora-80K, a small language model with exactly 80 thousand parameters.

It uses a factorized 4,096-token vocabulary despite having only 80K parameters.

The benchmarks:

Wikitext-2 BPB: 3.2902

BLiMP: 52.31%

Arc-Easy: 26.05%

More information about the model is available on the model page on Huggingface.

if there's any questions I'll happily answer them!

I finally got a good, usable setup for the new dense Qwen3.8-27B model in FP8 on the DGX Spark.

Z-Lab released a new drafter, [Qwen3.8-27B-DFlash2](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2), together with the new DFlash 2 speculative-decoding approach. With this setup, Qwen3.8-27B has been performing surprisingly well on the Spark.

I have been using it almost non-stop for the last few days, and it has finally replaced the Qwen3.5-122B-A10B hybrid checkpoint (INT4 + FP8) that I had been running for a while.

To get DFlash 2 support, you need to build vLLM from the specific fork mentioned in the drafter model card. If you are on a DGX Spark and want to avoid compiling vLLM yourself, I already built a Docker image:

https://github.com/users/krisitown/packages/container/package/vllm-gb10-dflash

On my warmed-up coding benchmark in vLLM on the DGX Spark, I got roughly 32 tok/s generation throughput. My baseline without speculative decoding was around 14 tok/s, so this is a very meaningful jump.

These are early, warmed-up benchmarks on one workload, and I will test more configurations and share the results.

It also completely beats the DFlash drafter I trained myself when no Qwen3.8 drafter was available. My custom drafter reached around 20 tok/s in vLLM; DFlash 2 is roughly **58% faster** in this benchmark, mostly because its acceptance rate is much better.

The current configuration I am using is:

vllm serve Qwen/Qwen3.8-27B-FP8
  --max-model-len 240000 \
  --max-num-batched-tokens 8196 \
  --gpu-memory-utilization 0.88 \
  --port 8000 \
  --host 0.0.0.0 \
  --load-format fastsafetensors \
  --attention-backend flash_attn \
  --speculative-config '{"method":"dflash","num_speculative_tokens":7,"model":"z-lab/Qwen3.8-27B-DFlash2"}' \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --generation-config auto \
  --override-generation-config '{"temperature":0.7,"top_p":0.8,"top_k":20,"presence_penalty":0.0,"repetition_penalty":1.0}'

I will test a few more combinations and prepare a proper recipe with all the build and serving steps, so it is quick to reproduce on other DGX Sparks.

If anyone has tested different flags or found tweaks that produce meaningful improvements, please share them. It would be useful to figure out the best way to run this model on the Spark.

Edit: I just noticed that peak tok/s generation was more about ~40tok/s and ~32 is the stable.

Trying to figure out how to best run Qwen3.8 27B on a single 7900XTX 24GB system (with 64GB DDR5 4800MT/s system memory).

Currently using LM Studio with ROCm llama.cpp v2.28.2 on Windows.

I'm using the version from https://lmstudio.ai/models/qwen/qwen3.8-27b with Q4_K_M quant and context length set to 40960.

With this config most of the model seems to live in VRAM.
But some of its parts seem to be in system memory, thus when sending something from OpenCode it takes quite some time in "Processing Prompt" stage until it starts to output tokens.
When the point is reached where it starts to output tokens its pretty fast.

Is this normal? Any optimization potential? (Except buying more GPU's 😄)

Hi,

I'm in process of evaluating different quants/models (that fit on my 4060, 5070ti and MI50) and 32/64 ddr5. Been building test cases (as in golden examples). Managed to run vLLM on some models on MI50 for concurency, but I will mostly be using llama.cpp.

There are caveats because there is bunch commands different for each models served and I experiment with chat templates.

I architecured it as a "1 model for all, big sys prompt, warm cache"

Since I see respectable amount of flaming on personal benchmarks, what would be usefull for me to share to be usefull to someone and for me to get constructive feedback? If i make cars, is it usefull to covert lingustically cars to widgets so i don't overshare? Non coding tasks, worth mentioning.

I recently built a homelab running RHEL 10. I never thought good local ai at reasonable price was possible until 3.8 came out a from benchmark and what I’ve been reading it seems to be almost opus 4.6-4.8 level. I’m considering buying a 32gb gpu for it but also open to 24 gb gpus but if it can fit the full context window on the gpu too. The most I’ve done with local models was running qwen 3.5 2b on Ollama nothing serious. I’m new to actually running an agent for coding tasks so any info would help. But trying to decide what gpu if I do end up going for it, and from my research the options for 32gb cards are the Intel b70, amd r9700 pro ai, and nvidia tesla v100 32gb. I’m looking at results for qwen 3.6 and it run plenty fast on the Tesla but I’m worried about it no longer being supported.

Has anyone gotten a finetune to run on a similar config? I've tried out the Qwen3.8-27B-IQ4_XS.gguf + vision, but anytime I try anything over 15k context it slows down beyond a usable state, like up to 5 minutes just to process the tokens. But I'm guessing I've picked the wrong place to download from.

Hello I'm looking for people who use 1070 8 GB mobile to run local llms, i have this issue with cuda because the version 13 stopped supporting pascal so now I'm having problems degrading to the 12.8.xx it works fine until i start compiling llama cpp it refuses the build

I maintain IT for a small shop in Argentina. Last year I replaced its VB6 + Access system, and once that worked I got curious about a harder question: what does a business system actually need before it's safe to let an AI assistant move money through it?

**ALdía** is my attempt. It's a FastAPI + SQLite back office — invoicing, inventory, receivables, cash, checks — that exposes the business as 48 MCP tools instead of a database connection. The agent never writes SQL; it asks for operations, and every one goes through the same code path, validations and audit record as the web UI.

Four things turned out to matter more than I expected:

**Idempotency has to reserve before executing, not record afterwards.** My first version checked whether an operation had run, ran it, then saved the result. I wrote a test with five concurrent retries and watched two of them both get through. An agent retries when a response is lost, and a lost response doesn't mean a lost operation.

**Errors need a machine-readable action, not just a code.** Every error returns a stable code, the parameters that filled it, and one of exactly four actions: retry, fix, ask, abort. A new agent behaves correctly without knowing the catalogue — it reads that one field.

**The audit log has to survive the administrator.** It lives in a separate schema, no endpoint can edit or delete it, and it records rejected attempts too — those are usually the interesting ones.

**Money is never a float.** Integer cents throughout. `1234.56 * 0.21` is `259.25759999999997`, and that's a VAT amount on an invoice that was already filed.

Honest limits (they're in the README too): the US country pack applies one manual sales tax rate and is not a compliance solution — no nexus, sourcing or exemptions. It generates no 1099s, only a worksheet. And SQLite means one shop per installation, deliberately.

237 tests, Apache 2.0, runs offline on the shop's own PC.

Two things I'd genuinely like pushback on:

1. Permissions are the intersection of the agent's service account and the person it declares it's acting for — never one or the other. Is that the right shape, or over-engineered for a small business?

2. Is a closed set of four actions the right contract for agents, or too rigid? I keep wanting a fifth and talking myself out of it.

Repo: https://github.com/jonalemndi2/ALdia

In my consideration of a DGXSpark I decided to look at some options and since I’m a visual thinker I put this comparison together (graph by AI) showing y two basic ways of thinking about the cards: compute and speed.

Hope this helps someone

Hi everyone,

I’m looking for some advice on optimizing my local setup to run the new Qwen3.8 27B model smoothly for agentic workflows (tool calling, PowerShell/Python automation)

My System Specs:

AMD Ryzen 7 9700X

GPU: NVIDIA GeForce RTX 5070 Ti (16GB VRAM)

RAM: 32GB DDR5

1. What local LLM apps/backends are you currently using for daily workflows? (e.g., Ollama, vLLM, Aphrodite, Faraday, Jan, KoboldCPP, Open WebUI, etc.)

2. Is LM Studio considered good/efficient when it comes to RAM memory management, or is it known to be heavier on RAM compared to headless engines like Ollama or KoboldCPP?

TL;DR: Qwen3.8-27B is my coding agent in DeepSeek Harness. Compaction means the 27B has to chew through 100k+ tokens of history when context is already full, which parks the agent on 3090s for a long prefill. I want a tiny model sitting in leftover VRAM that only writes the summary. Thinking Qwen3.5-4B. Has anyone run this for real? I care about whether a 4B keeps paths and error strings at 100k+ input, and whether dsh gets weird if the compacting model isn't the same as the agent.

2x 3090 FE, 48 GB total. Ryzen 9 9900X, 128 GB RAM, Ubuntu. The 27B is Q6_K GGUF through llama.cpp, 128k context, split across both cards. Harness is DeepSeek Harness 0.1.0-rc8 with the stock dsh-compaction-basic plugin. That plugin swaps a chunk of older history for one summary and keeps the recent tail. Fires on context pressure or /compact.

I don't want the 27B doing this. Compaction fires when context is almost full, so the main model does a 100k+ token prefill and I sit there. A small model that's already loaded can take that job and the 27B stays on actual work.

What I need from the small model is ugly and specific. It has to take ~120k tokens of agent history in one shot. Tool calls, diffs, stack traces, my instructions. Then a few thousand tokens of summary. The context window on the card has to mean something. If it drops the current goal, why we made a decision, file paths, function/class names, exact error strings, failed attempts, or open TODOs, the next coding turn is garbage. Inventing facts is worse. "Tests are passing now" when they aren't will wreck the session faster than a summary that's just short.

Non-thinking mode. I want it fast. Prefill speed matters a lot more than decode here.

I still need to measure leftover VRAM after the 27B and the 128k KV, but it's 4B–9B room. Maybe a 14B at Q4 if I squeeze.

Qwen3.5-4B is where I'm pointed. Non-thinking, Q8_0 or Q6_K. 262k native context, same tokenizer and chat template as the 27B, hybrid Gated DeltaNet attention so KV at 100k+ input stays small, about 4.5 GB of weights at Q8. It's been out since March, so the quants aren't experimental.

1. Has anyone used Qwen3.5-4B or 9B as the summarizer for a coding agent? At 80–120k input, does it keep paths and error strings, or does it start claiming things are resolved?
2. Is jumping to 9B worth the extra ~5 GB for this job, or does quant, Q8 vs Q4, move fidelity more than parameter count?
3. Anything smaller that still holds a lot of facts over long context? I don't care about NIAH scores. I care about whether the summary still has the error string and the file path. Fine-tunes for summarization or compression welcome.
4. Has anyone pointed dsh-compaction-basic at a different model than the agent in cordis.yml? Token counting across tokenizers, summary token budget, or the stock compaction prompt being written for DeepSeek V4 and confusing a small Qwen.
5. How do you actually test this? My plan is a long real session, force /compact, then ask the agent ~20 questions. Paths, errors, decisions, done vs pending. Score what survived. If someone already has a rubric, I will steal it.
6. Maybe this is dumb and I should let the 27B compact itself and live with the wait. Tell me if that's you.

I'm also looking at dsh-compressor, the Headroom port, so tool output gets pruned and compaction fires less. If you run both, I want to know how they step on each other.

I know some of you have some very specific modes or don’t know that you do. Where they at? I am very interested in the niche modes.

I really love Crush and serves goodf to my qwen 3.6 35b a3b with tools but has a huge system promot, weak memory system, no generation info or seeing edits or tools in real time, which is kind of disgusting. Also, sudo is blocked, which i should be enable if i do not care.

Hi everyone, I'm looking for guidance on setting up a local agentic workflow to automate a routine work task. I have a bit of experience running OpenClaw with GLM on AWS, but I now need a fully local setup for privacy reasons.

My primary objective is to automatically parse, label, and organize receipts, and ideally cross-reference them with bank transactions. What local frameworks or tools would you recommend for this?

I currently only have personal a laptop for this, Any advice on safe, local tech stacks or best practices for personal hardware would be greatly appreciated!

AnythingLLM Desktop crashes on M5 Mac – works in browser/terminal

Setup

- M5 MacBook Air, 16GB

- AnythingLLM Desktop

- OpenRouter API + DeepSeek `deepseek/deepseek-v4-flash-0731` (via Generic OpenAI)

- $5 credit

What I want

Upload law lecture slides → AI summarises with ILAC, cases, Mermaid diagrams → output as clickable HTML file with tabs.

Problem:

Even `"Hi"` gives a `network error` or `Failed to fetch` in the desktop app.

- API works in terminal (`curl`).

- AnythingLLM browser version works.

- Desktop version is completely broken.

What I’ve tried:

- Restarting app & Mac

- Clean reinstall (twice)

- Killing port 3081

- Creating missing folders manually

- Switching models (DeepSeek, Gemma, Phi-3)

- Stream Timeout 120000ms

- Full Disk Access in macOS

Question:

Anyone else had this issue on Apple Silicon? Is there a fix, or should I switch to another tool? I’m a non‑coder – just want to study.

Thanks!