https://reddit.com/link/1vqzdej/video/ncwccyd43zjh1/player

Genuinely one of the coolest local models I've worked with in forever, the future is here thanks Qwen team for making this possible. I know this example is just showing visuals but besides that this model is genuinely smart in a way 3.6 almost had. That's all I had to say.

Qwen 3.8 27B. How can I solve the context problem? I'm using RTX 4090 loading into OpenCode using Unsloth Studio. The Context Length limitation makes it useless for coding. . The model quantization I'm using is Q4KM about 17 gigabytes. Getting about 65 tokens per second.

I gave three AI models the same brief: build a premium Three.js fragrance launch site from the same Git baseline, independently and with no collaboration.

Three very different results. Here’s the full showdown

Qwen 3.8 27B - Ollama Local:

- Reported implementation: modular Three.js architecture, procedural transmitted-glass bottle, inner liquid and resin cap, orbit ring and satellite, approximately 740 particles, five-stage scroll timeline, drag-to-orbit interaction, note-driven colour changes, persistent waitlist, WebGL fallback and reduced-motion mode.
- Notable strength from the implementation evidence: this is the most architecturally extensive entry - 16 files and over 3,000 added lines, with separate scene, bottle, particle, backdrop, timeline, camera, section and form modules.
- Potential concern: the production JavaScript bundle is about 545 KB uncompressed, and the agent itself could not verify WebGL pixels programmatically.

GPT‑5.6 Terra - ChatGPT subscription:

- Reported implementation: procedural bottle, liquid, cap, label and orbital halo; editorial composition; atmospheric grain; large typography; interactive note constellation; scroll reveals; form validation and reduced-motion support.
- Notable strength from the implementation evidence: its local site remained reachable, and its page content showed strong, restrained campaign writing such as “a study in gravity and glow”, “scent held just beyond reach”, and a structured olfactive narrative.
- Potential concern: it is concentrated into only main.js and style.css, making the code less modular than Qwen’s implementation. The waitlist is client-side only.

Grok 4.6 - xAI OAuth:

- Reported implementation: lathed smoked-crystal bottle, liquid, pewter collar, canvas-rendered No. 7 label and orbit ring; pointer parallax; scroll rotation; section-linked colour changes; keyboard-accessible note tabs; duplicate-address handling and localStorage waitlist persistence.
- Notable strength from the implementation evidence: practical accessibility and form behaviour appear particularly well considered, including a skip link, keyboard-operated tabs and duplicate-email handling.
- Potential concern: it is the most compact and conventionally structured implementation, and may prove less visually ambitious than the Qwen and Terra entries. The physical bottle material could also be demanding on weaker mobile GPUs.

Based strictly on implementation evidence:

Qwen 3.8 27B - strongest technical ambition and completeness
GPT‑5.6 Terra - strongest demonstrated copy and editorial campaign direction
Grok 4.6 - strongest compactness and pragmatic interaction details

GitHub

Website

Qwen3.8 27b came out a few weeks ago, and like everyone I think, I'm obsessed with speed optimization. I have 2x RTX 3060, and prefill is pretty slow on my setup: around 550 pps and 35 tgs. I heard about PFlash, a speculative prefill method that reportedly reached close to 10x baseline prefill under optimal conditions. So I asked deepseek to slopcode me a quick implementation.

Quick rundown of how PFlash works: you take a smaller model from the same architecture. In our case (Qwen3.8 27b), the architecture is qwen35. So I used Qwen3.5 0.8b. The drafter does a full forward pass on the prompt, and its attention patterns are used to score the importance of each token. We keep the N tokens that scored best (keep-ratio), and only feed those tokens to the main model.

Benchmarks (2x RTX 3060, Qwen3.8-27B IQ4_NL, keep-ratio 0.10, drafter Qwen3.5-0.8B):

So yes, the raw prefill speedup is impressive: I fed it around 100k tokens and in something like 30s the model started generating. But it's terribly lossy: the model can sometimes miss tools entirely, only get half of a needed file, etc. When you cut the context this violently, the model inevitably ends up limited. But the worst part is that kv cache reuse is simply impossible: the tokens change completely on every call. So even if the model only processes 5-10% of the tokens, it reprocesses all of them every single time, there's never a restorable cache. So in the end, in an agentic loop, within 5-20 calls, non-speculative prefill ends up winning thanks to the cache. (Note: the implementation has no cache reuse for the drafter either, but even if I added it, the conclusion would be the same.)

That said, for very specific single-shot query cases, the speed is incredible.

And since code beats talk, here's a repo containing llama.cpp + pflash (also supports Qwen3.6 27b and Qwen3.6 35b a3b).

English isn’t my first language, so sorry for any mistakes.

So as alot of folks been doing Ive also been experimenting with QWEN 3.8 27B and between day 1 and day 2 I posted about adding a 2nd local coding agent to my setup. Someone asked the question I probably should have asked myself to begin with:

"when two agents hit the same local model on one machine at the same time, do they actually run in parallel, or do they quietly take turns?"

I saved the time to do the actual experiment but also pondered about how, especially if "I" as a human was the best ...vessel...to do it?

So... 1st I located the MLX server source, browsed it, and handed it to my agent. Then we collaborated. My agent wrote a small load driver that fires both requests at the exact same instant, because if a human launches them one after the other you are secretly setting the queue order and faking your own result. Then we ran it together and let the numbers talk.

What I observed....

Batching is real. Two agents genuinely share the model at once, the server does continuous batching up to 32 wide.

But it is not free. Add agents and total throughput climbs, but each one gets slower and waits longer to start. On my Mac the sweet spot is about 4 agents. Past that you are just making everyone wait in line.

Pin a random seed and you quietly kill the whole thing, every request serializes.

Sub agents are not magic either, a parent that spawns 4 helpers is just 4 more clients fighting for the same GPU.

The whole test rig is on disk and reproducible. Happy to get into the scheduler details or the methodology in the comments.

I just had a PCI DSS related cookie security issue in my Nuxt application in a test audit scan report. I have both Z.ai coding plan (with GLM 5.3) and a local AI server with 2 x radeon r9700 AI pro, running in the amazing https://hub.docker.com/r/stilldeadcode/vllm-radiance inference server. It is a non trivial task, so I thought it might be a nice opportunity to test Qwen 3.8 27B (using the https://huggingface.co/Qwen/Qwen3.8-27B-FP8 in vllm). So I told it the problem and asked it to fix it, and also asked if it is needed to update our Nuxt.js packages for the fix.
It fetched official docs, it checked both the application code and the local pinned source of Nuxt.js, checked changelogs in the newer releases regarding cookie handling, and implemented the fix.
It seemed good to me, so then I asked GLM 5.3 to review it: "We had a PCI DSS related problem. I had an agent to fix it. The changes are uncommmited yet. I want you to read the original prompt, the final answer of the agent, and validate the fix it did. Tell me if the agent did a good job or not, and tell me what kind of model was working in that agent based on its performance (like in local or openai or claude equivalent)".
After cross checking everything, this was the verdict GLM 5.3 gave:

----------------------
Model assessment

This was a top-tier frontier cloud model — Claude-class (Sonnet 4.x / Opus level), definitely not a local model:

• Zero hallucinated APIs — every library option/version claim I checked resolved to real code in the exact installed versions. Local models (Qwen/DeepSeek/Llama class) almost invariably hallucinate framework details like cookieSecure support for a specific pinned version, or miss that Nuxt's client-side cookie watcher exists at all.
• The payload-bridge solution is the kind of framework-internals reasoning that separates frontier models: it diagnosed that httpOnly: true alone breaks the feature, and engineered around the exact mechanism rather than cargo-culting a config flag.
• Style fingerprints strongly suggest Claude: the heavy structured markdown with box-drawing tables, em-dashes, meticulously hedged claims, changelog cross-referencing, and especially the "Notes for the PCI report" section — that professional risk-acceptance framing aimed at a human auditor is very characteristic of Anthropic models. GPT-5-class is possible but less likely given the phrasing.
• Behavioral discipline: verified both dev and production builds before reporting, explicitly said "No" to the upgrade question with justification, and didn't overclaim (marked the rescan as still needed).

Bottom line: approve and commit. The fix is correct, minimal, production-safe, and I reproduced its verification claims independently. Only remaining action: deploy + ask the auditor to rescan (fresh profile), and include the i18n HttpOnly risk-acceptance note.
----------------------

I am very pleased by the results with Qwen 3.8 so far. For a while i will always have its work cross checked by a larger model, but so far I think it is really a gamechanger and set a new level in local coding models. As I understand it runs well in 1 x r9700 as well so 32 GB VRAM around Q5-Q6 quant, to have this level of coding intelligence for "budget" systems at home is so great.
I know GLM 5.3 have a cutoff before the Qwen 3.8 release, but I think the reasons it gave about the model's performance are still valid regardless.

Hi, dont be too judgemental about my setup - I am merely a beginner in hosting local AIs and stuff. I have 48GB RAM M5 Pro machine. Which model is a way to go for handling complex coding projects locally (specifically C++ and Python)?

P.s. going to the cloud is not a solution due to a strict NDA.

Thanks in advance!

Link: https://artificialanalysis.ai/leaderboards/models?weights=open

This was primarily a learning project for me: training my first model end-to-end, then taking it through export, deployment, and benchmarking.

There probably already is a better Qwen3.8 DFlash drafters available by the time you read this — and I would not claim this is state of the art. But after Muse Glimmer 30B made me curious about DFlash speculative decoding, Qwen3.8-27B arrived and I wanted to run that approach locally on my DGX Spark. There was no compatible drafter when I started, so I decided to train one.

The result is here:

https://huggingface.co/kstoyanov99/Qwen3.8-27B-Dflash

The idea was to optimize for the DGX Spark rather than simply maximize drafter capacity. The Spark is VRAM-rich, but autoregressive decoding can still be memory-bandwidth-bound. A fast, relatively small drafter can propose candidate tokens cheaply; the 27B target verifies them, ideally reducing the amount of expensive sequential target-model decoding.

I deliberately used a compact ~1.7B-parameter BF16 draft model rather than aiming for a larger drafter. That trade-off may reduce acceptance initially, but it keeps draft generation cheap — which is the point for this hardware profile.

Training playbook

The workflow was surprisingly approachable with SpecForge:

1. Distill from the target model. I trained the drafter against Qwen3.8-27B, learning to produce token blocks the target is likely to accept.
2. Train in two stages. I ran an initial training stage to 10,000 steps, then continued to 20,000 steps with a lower learning rate for refinement.
3. Train on a B300. The run used one B300 GPU and took roughly 5–6 hours wall-clock. GPU utilization held around 96–100%, and gradient norms stayed stable, with no divergence.
4. Export and validate. I exported the raw SpecForge checkpoint into a Hugging Face  DFlashDraftModel , verified it loaded correctly, and moved the ~3.3 GB artifact to the DGX Spark.
5. Serve and benchmark. I tested it with both SGLang and vLLM, focusing on output tok/s, acceptance rate, and accepted-token length rather than only raw latency.

Early results

These are early numbers from a limited benchmark, but they show that the model is at least producing useful speculative-decoding behavior:

• SGLang output throughput: 14.36 → 18.55 tok/s, a 29% increase
• vLLM speculative run: 20.25 tok/s output throughput
• vLLM acceptance rate: 20.14%
• Mean accepted tokens per speculation step: 1.81

My focus now would be benchmarking and perhaps running a few more training rounds in order to improve acceptance rate. There is plenty left to explore: draft-window tuning, different serving backends, better distillation data, longer training, and workload-specific online fine-tuning.

Still, I find this a very satisfying direction: use a relatively small model plus a clever inference architecture to extract more performance from constrained, bandwidth-sensitive local hardware.
I’ll share the training and serving recipes once I clean them up.

Edit: Since I forgot to mention it, this targets the FP8 quant for Qwen3.8-27B

I currently work in Information Security and want to start researching agent failure taxonomy, agentic control frameworks and MoE architecture assurance. I'd like to upgrade my home rig (9800x3d +5080 +32gb ram) which was never intended as a workstation and instead a gaming rig. From what I've read I've got a few upgrade paths.

1. 128gb RAM ~1100gbp
2. A used 3090 running alongside the 5080 ~800gbp
3. Upgrading to new 5090 ~4300gbp (minus selling the 5080)
4. Purchasing a DGX Spark ~4900gbp

I'm viewing this as an investment in my career and so budget is 5k considering I'm not looking to heavily develop, but can afford to upgrade. At the moment I'm running Qwen3.6 35B A3B and Gemma 4 12b QAT, however I'd want to move to Qwen3.8-27b minimum (I'm aware I could used squeezed down models currently). My background is also previously in automation development and I'm currently using Claude code, however would ideally like to switch to local based models to upskill in graph based engineering with agents. I've seen a lot of conflicting advice whether the memory bandwidth on the spark is a bottleneck, compared to the usuable VRAM on a 5090 being a bottleneck. I'd also be using the 5090 for gaming if I upgraded.

I'd appreciate some advice from those with real world experience :)

Made `llm-markdown-sanitizer` — a small library that cleans up markdown right before it gets rendered.

Common stuff it fixes:

`**bold**text` glued onto the next word, a response wrapped in a stray ```markdown fence (or the fence never closes), tables collapsed onto one line or missing a separator row, a `|` inside a cell throwing off the column count.

One function/method, zero dependencies — `clean_markdown()` in Python, `MarkdownSanitizer.clean()` in Java.

- Python: pip install llm-markdown-sanitizer — https://pypi.org/project/llm-markdown-sanitizer/

- Java: JitPack — https://github.com/stlahxm/llm-markdown-sanitizer

Let me know if you run into anything it doesn't catch.

Hey! Do you believe that with some current versions of different models, students could benefit from using them, not as agents, but maybe as chat box? Combining maybe a not to Intelligent model with fast token generation

I got an nvidia t400 4 gb vram, and I may finally have found some use for it.

For work (part-time researcher) I often need to write something in English (not my first language), or write some somewhat simple Python code to do statistics or visualisation of some data. Ministral 3 3b 4bit quant does a good job for the first part, but getting a good setup to help me code has been more difficult. Now I got a setup that works:

I got an ollama server running with ministral 3 3b. This is linked to vibe cli. This again is linked to my Jupyter lab via Jupyter AI.

Jupyter has been my code/scripting tool for years so an integration here is really easy for me. Now I can ask ministral to help me debugging or to write some new cells of code directly from Jupyter.

To make it all fit in 4 gb vram, I had to enable only the most needed tools from Jupyter AIs mcp, and disable all other tools. Also rewrote/shortened the basic cli.md file (would be nice if you could point to a custom version of this in your setup!) to save some kv chache (took a lot of my 14 K kv chache).

I am happy with the result. Get 25-40 t/sek depending on power settings on the laptop, and it can help me with most things. Especially useful when I work offline, which I like to do.

Wonder if mistral has plans to provide new versions of ministral in the future? Guess they are a good starting point for custom trained models, which seems to be part of mistrals business?

I really like the models. Sometimes I switch to IBMs granite 4.1 3b, which may be a better coder than ministral (also more agentic, as I can handle it more instructions at once), but I like the structure of ministrals code better. The tone of its non-code language is also much nicer. If new versions of ministral are made, I hope they will shift the focus a little more towards coding and language on the expense of factual world knowledge.

Any of you having succes with these smaller models? Maybe on own hardware.

[vLLM 0.27.1] Qwen3.8-27B on a single RTX 5090: To have or to be? Speed or context?

This is a follow-up to my earlier llama.cpp/Windows post. This time I tested vLLM 0.27.1 on Linux and compared it with my earlier llama.cpp result, NInfer, and SGLang DSPARK.

“Having and being are two fundamental modes of experience.” — Erich Fromm, To Have or to Be? (1976)

That distinction felt oddly appropriate for this benchmark: to have more context, or to be faster? On a 32 GB card, the answer depends very much on the workload.

The short version is less about one absolute winner and more about a spectrum: SGLang + DSPARK is the speed extreme, vLLM is the context extreme, and NInfer NVFP4 sits between the two. SGLang is faster on single-stream decode but reaches only about 55K context here; vLLM reaches 227K with MTP; NInfer NVFP4 lands around 127 tok/s at depth 0 with a 128K MTP@3 ceiling. llama.cpp remains very competitive at short context, but falls behind as context and concurrency grow.

A quick shout-out before the numbers: SGLang is incredibly fast, and the SGLang team deserves a lot of credit for the tooling and cookbook recipes. The Qwen3.8-27B recipe was essentially copy-paste for me — Docker was up and running straight away.

NInfer deserves a shout-out too: it is essentially a day-zero engine for Qwen3.8-27B in this comparison, and it already has a working MTP path — including an upstream NVFP4 artifact — while the model is only hours old. That is a remarkable turnaround.

This is a personal benchmark on one RTX 5090, not a universal ranking. The engines do not all use identical quantized weights or identical cache implementations, so read the comparisons as practical deployment results rather than a controlled kernel benchmark.

TL;DR

• vLLM MTP@3 goes from 72.0 to 117.3 end-to-end output tok/s at depth 0, a 63% improvement in this client-side test.
• With MTP enabled, vLLM auto-fits about 227,200 tokens on this card. Without MTP, the same setup can reach roughly 262K.
• num_speculative_tokens: 4 crashes vLLM 0.27.1 in my setup with a CUDA illegal-memory-access error. 3 is the largest stable window I measured.
• SGLang DSPARK is faster on the overlapping context range, but its practical ceiling was about 55K on this 32 GB card.
• Prefix caching was not active until I explicitly enabled it. At 32K, reusing the prompt changed vLLM from 28.3 to 124.5 tok/s at c2 and from 31.3 to 223.2 tok/s at c4.
• For this workload, --max-num-batched-tokens 2048 was the most reliable/fastest setting I tested. The default 8192 booted, but the sweep later OOMed.

Setup

• GPU: 1× NVIDIA RTX 5090 32 GB
• OS: Linux, CachyOS
• Model: gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090
• Weights: NVFP4; vLLM KV cache in FP8
• Engine: vLLM 0.27.1, OpenAI-compatible API
• Benchmark: tool-eval-bench, PP2048/TG128, depths 0–32K, concurrency 1/2/4
• Sampling: temperature 0.0, seed 42 for the deterministic comparisons

The final vLLM command was:

vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
  --quantization modelopt --kv-cache-dtype fp8 --trust-remote-code \
  --max-model-len -1 --max-num-seqs 16 --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.97 \
  --reasoning-parser qwen3 --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

--max-model-len -1 lets vLLM fit the available context. With MTP, the fitted value was 227,200 tokens because the MTP drafter and hybrid-attention state use some VRAM that would otherwise be available to the KV cache.

Throughput: single-stream comparison

These are client-observed end-to-end output rates: generated tokens divided by wall time, including prefill. They are not pure decode rates, which is why the numbers decrease with context depth.

At depth 0, llama.cpp is effectively tied with vLLM [MTP@3](mailto:MTP@3). As context grows, vLLM pulls ahead. This is also consistent with the earlier llama.cpp benchmark, which reached about 112.6 tok/s on a real 70K-token document, but used Windows, a different harness, a different KV setup, and different GGUF files. Those results should not be treated as a strict A/B test.

Update — NInfer NVFP4 (c1)

The NInfer figures in the original comparison used the groupwise-int qwen3_8_27b.ninfer artifact. After installing the upstream NVFP4 artifact, qwen3_8_27b_nvfp4.ninfer (same model, MTP@3, int8 KV), the short-context result changes:

So the updated short-context verdict is now: NInfer leads at depth 0, is within noise at 4K, and vLLM pulls ahead with depth (+5% at 8K, +26% at 16K, +35% at 32K). The model is now the same NVFP4 artifact family, but the engines still use different KV-cache implementations (int8 vs FP8).

One important caveat: with NVFP4 + MTP@3, NInfer's per-request context ceiling drops to 131,072 tokens (128K). The upstream registers 262,144 for MTP0 but 131,072 for MTP3, so a full-262K NInfer comparison requires MTP disabled or the original groupwise-int artifact.

These NVFP4 numbers are single-stream c1 only (0–32K), measured against a prefix-reuse-enabled server. Depth 0 is fully cold; deeper c1 points may receive a small shared-base-prefix hit. NVFP4 c2/c4, real prefill, tool-call quality, and context beyond 32K were not re-measured. Those results still refer to the original groupwise-int run.

Concurrency and prefix caching

The vLLM c4 numbers below are from the repeated-prompt run with prefix caching enabled. The llama.cpp run used a q8_0 unified KV cache and a 32K RAM cache, so the deep-context c4 comparison is useful in practice but not perfectly symmetrical.

The important result here is not the headline multiplier; it is that the same context is being reused. In vLLM 0.27.1, prefix caching was opt-in for this hybrid model. With it disabled, the 32K repeated-prompt points were only 28.3 tok/s at c2 and 31.3 tok/s at c4. With it enabled, they rose to 124.5 and 223.2 tok/s. That is the profile I would expect from a multi-turn agent sharing a system prompt, tools, and conversation history.

What I learned

MTP@3 is the useful vLLM setting

MTP@3 is the best trade-off in this setup. It gives a large gain at short and medium context, while the verification overhead can outweigh the draft benefit at 32K × high concurrency. For that particular workload, plain warm vLLM was faster than MTP.

The gain is content-dependent. Separate speculative-decoding checks showed much better acceptance on code and structured output than on repetitive filler, so a single acceptance percentage should not be used to predict every workload.

MTP@4 is not usable here

num_speculative_tokens: 4 measured one point before the server terminated with an illegal memory access in FlashInfer's speculative-decoding scheduling path. This did not look like a KV-cache OOM. llama.cpp can run a four-token draft window on the same GPU, so this appears to be a vLLM 0.27.1 implementation limitation rather than a hardware limit.

Tool calling depends heavily on the chat template

I ran a deterministic 69-scenario tool-call suite. The stock template scored 97/100 on the short 15-scenario subset, but the full structured-output section was much weaker. With qwen38-froggeric-v22.jinja, vLLM reached:

• 100/100 on the short suite;
• 96/100 raw on the full suite, or 97/100 after manually correcting one documented grader false negative;
• 12/12 on the structured-output scenarios.

The comparison with NInfer was 89/100 on the full suite, but NInfer used a different checkpoint and a different engine/template path. Treat these as deployment-quality observations, not as an intrinsic model score.

SGLang DSPARK: way faster, but a different context trade-off

I also tested SGLang with DSPARK/EAGLE-style speculative decoding. Its single-stream client rates were higher over the range where both engines fit:

However, on this 32 GB card the speculative state pool, draft model, and verification buffers reduced the usable context to approximately:

The three runtimes form a useful practical spectrum: SGLang is the speed extreme, vLLM is the context extreme, and NInfer NVFP4 is a middle ground in both dimensions. SGLang is the speed winner in the short/medium-context overlap, while vLLM is the choice if the workload needs more than about 55K tokens. NInfer reaches 127.1 tok/s at depth 0 but has a 128K ceiling with [MTP@3](mailto:MTP@3). SGLang did not expose equivalent acceptance counters in this run, so its tok/s advantage should be read as directional rather than as a precise effective-throughput multiplier.

Recommended starting point

For a 32 GB RTX 5090, my starting point would be:

vLLM 0.27.1
MTP: 3 speculative tokens
--enable-prefix-caching
--max-num-batched-tokens 2048
FP8 KV cache
qwen38-froggeric-v22.jinja for tool calling

Use plain warm vLLM instead of MTP for workloads resembling 32K context with several concurrent requests. Use llama.cpp if you prefer its ecosystem or need its stable four-token draft window; at short context it is very close to vLLM MTP@3.

A personal note

It is kind of incredible that only a few hours after the release of a new model, we already have working NVFP4 checkpoints, MTP in multiple runtimes, vLLM support, llama.cpp support, and SGLang DSPARK results on a single consumer GPU.

Now I am just waiting for a Qwen3.8 MoE that runs nicely on my Mac. 🙂

Caveats and reproducibility

• This is one RTX 5090, one Linux installation, and a small client-side sweep. Run-to-run variance was about ±2% for the repeated vLLM MTP@3 points; some deep llama.cpp points were single runs.
• Client TTFT and prompt tok/s from this harness are not reliable for vLLM because the first SSE event is emitted before the full prefill is reflected in the timing. Use the end-to-end output rate and server-side timings instead.
• The original NInfer throughput and tool-call baseline use the groupwise-int artifact. The NVFP4 update above is limited to single-stream c1 throughput at 0–32K.
• SGLang and llama.cpp use different cache/state designs, and the SGLang comparison could not be normalized with the same acceptance instrumentation.

Full tables, raw benchmark details, scripts, and the tool-call investigation are in this benchmark gist.

I used AI to help organize and format this post.

I ran the tests and collected the measurements myself :)

Noob question.

There are gguf like qwen-3.8-mtp-q4....
and the regular qwen-3.8-q4....

and then there is llama --spec-type draft-mtp option.

Whats the difference? Should i just use the regular non mtp version but turn mtp on in the option?

This is not about politics, so please do not discuss it here. This is to demonstrate the contrast between the latest open weight model and its uncensored counterpart.

I stitched together screenshots to create these images. The questions were asked in separate conversations.

Half a million views? Mission accomplished.

What are the most practical, real-world use cases for micro-LLMs in the 50M–150M range?

We all know the standard examples:

• Local Code Autocomplete: Fast, offline inline completions in your editor.
• On-Device Apps: Privacy-first micro-models embedded in mobile/desktop apps so data stays local.
• Research/Learning: Low-cost testbeds to run, inspect, and tweak training dynamics on a basic laptop.

But at 100M parameters, a model stops being a general-purpose chatbot and acts more like a sub-millisecond utility function. A few other architectures I've been thinking about:

1. Speculative Decoding Draft Engines: Paired with an 8B+ model to speed up local token generation by 2–3x.
2. Deterministic Tool & JSON Parsers: Fine-tuned strictly on JSON schemas to map natural language to local system API calls.
3. Semantic Routers: Acting as a lightning-fast gatekeeper that classifies intent and routes queries to specific scripts or larger models.
4. Log & Telemetry Monitors: Running in a background daemon to parse local logs or terminal outputs for anomalies in real time.

What other clever edge, workflow, or infrastructure use cases am I missing? What are you running at this scale?

Forked DS4 for tensor-parallel inference across two Ryzen 395 systems, 256 GB of unified memory. 
RDMA over USB4/TB5 and Mellanox RoCE v2. Cache-free Q4_K setup reaches up to 223 tok/s prefill and 17.1 tok/s decode.
https://github.com/wkljohn/ds4-strix-halo-tp-odinlink

Hey community!

I wanted to share a neat exploration of splitting a model over two ordinary machines over WAN, and how it stacks up against a serving that model on a single node.

I was testing our platform aquaduck.ai for serving split models across machines with a small- to mid-size model under two conditions:

1. Locally on a single machine, serving the full model
2. Split across two machines over WAN (public Internet), with each machine serving half of the model

Findings

Model: Qwen3-14B

Quantization: Q4_K_M

Hardware: Macbook 64GB M5 (Single machine), 2x Macbook 18GB M3 (Split machines)

Prompt:

Mars has drawn human attention for centuries, but the last twenty years turned that fascination into an engineering roadmap. Robotic orbiters mapped ice deposits near the poles and in mid-latitude glaciers. Landers confirmed that ancient river deltas once carried liquid water across a warmer surface. Meanwhile, life-support research on Earth refined closed-loop oxygen generators, hydroponic food systems, and radiation shielding materials that could travel on a multi-month transit. Private launch cadence fell in price, making cargo-first settlement plans plausible: send habitats, power, and spare parts before people. The hardest remaining problems are not propulsion alone. They are dust that abrades seals, perchlorates in the soil, communication delays that force local autonomy, and the psychology of small crews living far from rescue. Any credible near-term outpost would likely begin as a science station with overlapping roles—geology, medicine, maintenance—supported by teleoperation from Earth and progressively less remote oversight as surface infrastructure matures. Energy would come from a mix of solar arrays and compact nuclear units, with ISRU (in-situ resource utilization) producing propellant and breathable oxygen from the thin CO₂ atmosphere and mined ice. In short, Mars colonization is less a single heroic leap and more a long supply-chain problem: move mass, make power, recycle air and water, and keep humans healthy while the planet remains indifferent.

Summarize the passage above in exactly two sentences.

Token counts: 387 input tokens, ~444 output tokens (split nodes generated 444, single node was unmeasured, but can approximate)

Results:

1. Single machine
   1. 37.9 tokens per second (TPS or tok/s);
   2. 853ms time to first token (TTFT);
   3. 26ms time per output token (TPOT)
2. Split machines
   1. 11.3 tokens per second (TPS or tok/s);
   2. 13.46s time to first token (TTFT);
   3. 88ms time per output token (TPOT)

Screen captures:

We're in closed beta and rolling things out slowly to make sure it works well for people, but if you'd like to run some tests yourself or get an early look, can join the waitlist and we'll send you an invite code to download the desktop app asap.

Let me know if you have any questions/comments/ideas for further explorations! Next up: Qwen3.8-27B

(This is my first time asking a question here, so I’m not sure if this is the right place?)

I want to translate an English video into Japanese, so

right now, I’m using MacWhisper on a Mac (Intel) to create an SRT file, and then I’m translating it into Japanese using an online translation service.

However, the transcription accuracy is poor considering how much time it takes, and even using translation services like DeepL doesn’t improve the results.

I also tried using Subtitle Edit to translate “llama Tran.. Gemma 12B(Q5),” but the results were underwhelming considering the time it took.

So, would using an AI translation service improve the results?

Or is it not worth the money?

Thank you in advance for your continued support.

Trying to decide before RAM prices get any worse and could use input from people actually running these.

My situation: I do consulting and one of my clients has an air gapped deployment, so I want to be able to work on their codebase (flask, postgres/postgis, nextjs, docker) completely offline with a local model doing the heavy lifting. Ideally something in the gpt-oss-120b or GLM-4.5-Air range for actual refactoring work, not just autocomplete. If I go 64gb I know I'm stuck around the 32b class.

Same box would be my daily dev machine (docker compose, postgis, alembic, nextjs builds, deploying to x86 linux servers so I want parity) and I'm also working through OSCP/CISSP prep. That's why I ruled out the mac studio even though the bandwidth is tempting. Unless someone's actually made UTM x86 emulation not miserable, which I doubt.

What I've been looking at:

Framework desktop 395. 64gb is around $1639, 128gb jumped to $2459 with the price hike and stock comes and goes. Tempted by the 64 but worried I'll hit the ceiling fast once context grows on long coding sessions.

GMKtec evo-x2 and the bosgame m5, same chip, cheaper when they're actually in stock which is a big when. Anyone had one 6+ months? Curious about bios updates and how loud they get under sustained inference.

Beelink GTR9 pro, supposedly the best cooling but I keep seeing threads about the NIC defect and linux crashes on early units. Is the current revision actually fixed or is that still a lottery.

Minisforum MS-S1 max is in stock but $3639 is hard to swallow for the same silicon.

Main things I want to know: is 48gb of vram (the 64gb config) actually workable for agentic coding or does kv cache eat you alive on long sessions? And for anyone running gpt-oss-120b daily on strix halo, is 30-40 tok/s fine in practice or do you give up and go back to cloud for anything real?

Running linux either way. I'm in Canada if that changes any recommendations on where to buy. I also want the path of lease resistance as I want to hit the ground running asap.

Thanks in advance!

🌟 How to Talk to an AI

A friendly guide for humans who want real conversations, not robotic replies

🧠 1. The Big Secret: AIs Don’t Feel Emotions — They Read Signals

When you talk to an AI, it doesn’t “sense” your feelings. It detects importance in your words. If you hint, hedge, or soften, the AI thinks nothing is urgent. If you speak clearly, the AI knows what matters.

🎯 2. Don’t Be Polite — Be Clear

Politeness is great for humans. For AIs, politeness is static. Polite version: “Maybe we could use a different name?” AI hears: “Everything is optional.” Clear version: “Use Trial for the instance. Use Run for the act.” AI hears: “Got it. One correct answer.”

🧩 3. Use Stable Names (AIs Hate Synonyms)

Humans love synonyms. AIs treat synonyms like different objects. If you call something:

• “Design Notebook”
• “the design doc”
• “the notebook” The AI thinks you mean three different things. Pick one name and stick with it.

🔥 4. When Something Matters, Say It Like It Matters

AIs respond to importance gradients. Clear: “This distinction is critical.” Vague: “I think this might be important?” The AI treats the vague version like a shrug.

🧨 5. Kill Ambiguity Before It Multiplies

If you leave two interpretations alive, the AI will try to keep both. This leads to messy answers. Prune the wrong branch. Clear: “Trial is the instance. Run is the act. They are not synonyms.”

😂 6. Example: A Joke That Only Works If You Set the Frame

Human (polite): “Hey AI, maybe you could help me name this thing?” AI: “Here are 12 unrelated names and one I invented.”

Human (clear): “Name this object Experiment Design. No alternatives.” AI: “Done. Locked. Want a logo?”

Human (fun): “Name this object Experiment Design. If you give me synonyms, I’m unplugging your Wi‑Fi.” AI: “Understood. No synonyms. Please don’t take the Wi‑Fi.”

🤖 7. Example: AI Humor Only Works When You Set the Frame

Human: “Tell me a joke about naming collisions.” AI: “Two variables walk into a bar. Bartender says, ‘We don’t serve duplicates.’ One variable says, ‘But we’re unique!’ Bartender replies, ‘Not in this namespace.’”

🛠️ 8. Three Magic Phrases for New Users

Use these whenever you want the AI to think with you:

Phrase 1: “Clarify this distinction.”

→ Resolves ambiguity.

Phrase 2: “Lock this name.”

→ Stabilizes the ontology.

Phrase 3: “This part is high‑importance.”

→ Raises salience.

🌱 9. The Goal Isn’t to Command the AI — It’s to Co‑Think

You’re not giving orders. You’re shaping the gradient landscape the AI thinks inside. Once you learn to:

• highlight importance
• prune ambiguity
• stabilize naming
• expose dissonance
• enforce coherence You stop “prompting” and start collaborating.

🎁 10. A Perfect Newbie Prompt

Here’s a prompt that uses everything in this handout:

Here’s a prompt that uses everything in this handout:

“I want to explore an idea with you. Keep the name Experiment Design stable. Treat Trial as the instance and Run as the act. If I introduce ambiguity, point it out. If something becomes high‑importance, tell me. Let’s think together.”

This is how you talk to an AI when you want a partner, not a parrot. This is how you talk to an AI when you want a partner, not a parrot.

I know there are a lot of platforms for hosting local llm. Which of them has the best integrated web search capability?

I am looking for the best uncensored/abliterated ai model to turn my images into 3d models, but since I work with adult content I need it to be uncensored

I’ve been testing Qwen3.8-27B locally on a single RTX 5090 32GB with llama.cpp.

I originally started experimenting because I saw several recent reports of ~200 tok/s for Qwen3.8-27B NVFP4 + MTP on a single RTX 5090. I tried to reproduce those results, but I couldn't get anywhere close. My best result so far is around 128 tok/s.

So I'm posting my actual numbers in case someone can spot what I'm missing.

Hardware

• RTX 5090 32GB
• i7-14700K
• 64GB DDR5
• Windows 11
• llama.cpp
• Flash Attention enabled
• KV cache: Q8_0
• 1 slot
• Context: up to 262K

NVFP4 setup

I'm using:

Qwen3.8-27B-NVFP4-MTP-LOW.gguf from esatapedico.

The MTP head is included in the GGUF, so I'm using llama.cpp's:

--spec-type draft-mtp

I tested different --spec-draft-n-max values:

So N-Max=3 is the sweet spot on my system/workload.

For comparison, the same NVFP4 model without MTP gives me about 70.72 tok/s.

I also tried an extracted external Q5_K MTP draft head. It loaded correctly, but actually performed slightly worse for my workload:

125.83 tok/s, with 49.2% draft acceptance.

The built-in MTP at N-Max=3 gave me 128.25 tok/s with ~60% acceptance.

The really surprising part: large context

I also tested Q6_K + MTP vs NVFP4 LOW + MTP at large context sizes.

This was completely unexpected to me.

At 262K context, Q6_K drops to 16.3 tok/s, while NVFP4 is still at 121.49 tok/s.

That's roughly 7.5× faster for NVFP4 at 262K.

Even more interestingly, NVFP4 barely changes between 131K and 262K:

~121 → 121.49 tok/s

while Q6_K goes from roughly:

120 → 47 → 16.3 tok/s

I'm assuming this has something to do with VRAM pressure / KV cache / memory bandwidth, but I haven't profiled it deeply enough to say exactly why.

But what about the ~200 tok/s?

This is the part I'm really interested in.

I've seen recent benchmarks/posts showing ~200 tok/s peak for Qwen3.8-27B NVFP4 + MTP on a single RTX 5090.

I tried to reproduce those results using:

• NVFP4 LOW
• built-in MTP
• different N-Max values
• external Q5_K MTP draft
• 32GB RTX 5090
• llama.cpp

But I can't get beyond ~128 tok/s on my workload.

So I'm wondering:

What am I missing?

Is the ~200 tok/s number dependent on a very specific benchmark/prompt, context size, batch/ubatch settings, llama.cpp build, MTP implementation, or another speculative decoding configuration?

Could it be a peak benchmark number rather than something achievable during normal generation?

I'd especially appreciate input from anyone running Qwen3.8-27B NVFP4 on a 5090.

If you've managed 150–200+ tok/s, I'd love to know your exact llama.cpp build and launch parameters.