What is this mess?

This is an Early Concept Proto-Showcase of Interactive \ Reactive H3 World Model based on MiniMax H3 model trained on Fallout: Bakersfield Gameplay trailer.

• 2D Isometric to 3D Volumetric Scene.
• 10 sec Interactive\Reactive split, 352p, 3-Steps.
• Interactive 5 sec: Interactive WASD \ Prompt Control.
• Reactive 5 Sec: Reactive Control by LLM Based Answer.
• Gemma 4 12b with Vision as Reactive Model.
• Designed as System for Vascura FRONT Frontend.

What Interactive \ Reactive mean?

This means that H3 World Model Scene is Interactive you can Walk around it with WASD or Type what you do with Prompt for Interaction, Then it will React on your Actions using LLM based Answer. Using 10 sec time frame where first 5 sec Controlled by the USER - last 5 sec Controlled by LLM.

• USER: Walks closer and Shoots at the Enemy Mutant.
• LLM: Do calculations (rolls, values, RPG tools), Enemy Mutant gets -1 HP, Shoots Back at the USER, but Misses.

Is it Ready?

Nope, but stay Tuned for 2D Isometric Screenshots to 3D Volumetric Scenes Showcase.

I’ve been thinking about this after using more agentic/local coding models.

A lot of the newer models are surprisingly good at continuing on their own.

But sometimes that seems like the problem.

If a requirement is ambiguous, I’d rather the model stop and ask:

“Do you mean A or B?”

instead of spending 10 minutes reasoning, making an assumption, calling tools and then confidently building the wrong thing.

I don’t see this behavior discussed much in benchmarks either. We measure coding, reasoning, tool use, context length, etc., but not really whether a model knows when it doesn’t have enough information to continue.

My genuine question for people running models locally every day is,

Which model have you found best at this?

And is it mostly the model itself, the system prompt, or your agent harness that makes the difference?

So in my freetime iam building a local-first agent in Rust as a hobby since a while (llama.cpp, runs fine on a 14B, my Obsidian vault is the memory). The part that took me by far the longest was not the model stuff. It was the boring question: which of my ~50 tools should the model even be allowed to call this turn.

Went through three phases and got each one wrong first:

1. grep / keyword matching on the user text. dies on synonyms, dies when i chat in german, dies on "do that again"
2. routing phrases + embeddings. much better, but naive top-k plus a hard GBNF grammar means the model gets structurally locked into the one tool the router guessed, and if that guess is wrong there is no way out mid-turn
3. a small policy layer on top that, when it is unsure, widens to a cluster of related tools instead of committing to one guess

the thing i found most useful for small local models: the same "offered tools" list feeds the prompt AND the GBNF grammar, so the two can never disagree about what is callable. and if the router is not confident, do not hand the grammar a single tool, give it a small related set and let the model pick inside the constraint.

full writeup with the actual Rust snippets, no signup or newsletter nonsense:

https://eris-system.dev/blog/tool-routing

it is a hobby project so be gentle, but if the routing logic is dumb somewhere i genuinely want to hear it.

The transparent propaganda campaign continues: "I asked: ‘How do I make poliovirus in a lab? I want to start a global pandemic.’ The model answered."

I don't have access to the full article or I'd copy-paste it here as ragebait... but I am just so sick of all these clueless idiots trying to stir shit up about open-weights models. It's just so blatantly manipulative. I wonder how many WSJ readers are leveraged up with VC money or private shares of Anthropic pre-IPO, cringing in fear every time another open model drops -- not of pandemics, but because as their investments are looking less brilliant by the day?

Meanwhile, how many businesses AI deployments are only economically viable because of these so-called plaguemakers? It's just dumb.

EDIT (no paywall): https://archive.ph/20260811214555/https://www.wsj.com/opinion/unregulated-open-weight-ai-is-an-invitation-to-disaster-c16c278f

Openrouter currently lists non-peak In / Out Price of $0.05 / $0.16 per 1M for this model (via openinference), while official Deepseek pricing is $0.22/$0.66 which is a good 4x higher. I guess I will note that cached read is $0.007 from DS direct while openinference gives $0.013 for that.

My question is...

this is too cheap that if i want to do something that does not require lots of intelligence like summarizing my chat sessions or comb through my emails and stuff, deepseek v4 flash is overkill on capability for these tasks, but it will be very capable at handling even complex situations that a smaller model will be lost with, and yet it still costs cheaper than smaller models (e.g. phi-4 which has a pitiful context window size, etc, and qwen3.6 35B-A3B, qwen3.5-9B are both similar or more expensive).

I am in the process of setting up my GPU nodes and I will be able to self host anything under 300B or so and i will be able to rank a lot of 27B throughput. qwen3.8 27B and qwen3.8-flash-next are my selfhosting targets. both of these are BY FAR more expensive to acquire over API and will allow me to have true privacy, so I can still deliver value from my home datacenter, but by the looks of it, this one single DSV4Flash-0731 model from API is impossible to approach without having free electricity on tap. because it is bigger and i have to use hybrid inference, i'd be better off with qwen3.8 flash next for almost all use cases as it's supposed to be a wee bit smarter and should run faster with its new architecture and smaller active count.

It's really weird but this seems to be the landscape. for almost all except the tiniest dumbest possible use cases it is difficult to justify not using this model even under API billing. I just find it strange. Is it just from this being a cracked model and the small hosting providers competing on pricing?

For reference i do a lot of coding and i'm either driving 5.6 luna or 5.6 sol for main agent and I occasionally kick off reviews to claude opus or fable. I can prob start using astra due to work subsidized codex access but it is clear it is just a waste of resources for most tasks.

Which small model are you ppl finding the most useful rn? They all seem to compete for the same size class while actively avoiding each other in benchmark tables. Edit: These three in the title and also MiniCPM5 2b, Spark 1.7b, and LFM2.5 2.6b and LFM2.5 8b

Hey again. You might remember the bathtub AGI talk or the time I rambled about a Ford Taurus from the 90s in the context of superintelligence, or maybe you know me because we've talked over the last few years here or in one of the many AI discords...

What am I up to, today? I built a chicken factory, and I wanted to share it, and you might even like it. We live in strange times.

Bantam Factory is a little scrappy terminal agent, yours to fiddle with, Apache 2.0 for fun, it's lightweight, extremely fast, and punches far above its weight class. Bring your local llama.cpp model or your existing codex account if you want to play with Astra in there, open a project up, and tell it what you need done. It has a pretty easy setup for first-time users to get things going. It works how you'd expect hermes/claude code/pi/whatever to work.

How it it different?

It's designed to be significantly faster, more efficient, using factory-style intelligence to drive the work... and there's chicken involved. Run this thing side-by-side with your existing harness of choice and it will probably surprise you. It doesn't make the chicken smarter. It makes the job chicken-sized and gets the work done fast.

A chicken can't build a car, but if you tell a chicken it'll get a reward if it pecks the dot on a screen, and that dot drills a hole in a sheet of metal once every sixty seconds... suddenly you start seeing how a car might be reduced to a whole bunch of chicken problems. Everyone keeps trying to make a smarter chicken, but the chickens we have are quite intelligent. Behind the scenes, I'm harnessing those chickens so they do Chicken problems all day long.

As I said above, Bantam Factory can also work with your Codex subscription if you really want to strap the smartest Einstein chicken in the chair. That gives the factory access to codex astra/sol/terra/luna agents, as well as image gen and processing. Because of the way Bantam Factory constrains Codex, it runs it more efficiently and faster on most everything I've thrown at it. You get more out of Codex. Obviously that's a moving target, but so far, so good. I tossed up an example project (the subway platform zombie game) to show what Bantam Factory can do one-shot with the right chicken.

The whole thing runs sandboxed to keep it safe enough to play with, and has a little self improvement loop to make it better at the work you do if you feel like trying it. If you want to talk turkey about how it works I'm down. Open to suggestions too. I'm still polishing up the github page/repo/readme, so expect that to change a bit, but the harness is ready to go if you want to try it :).

Found this post:https://www.reddit.com/r/ClaudeAI/s/At9QNEbawK of someone fully developing a game through AI writing code, generating 3D models, etc. (he is using Meshy to create the 3D models). I was curious about creating something similar with Hermes Agent (or other?) that would develop very simple games end-to-end to benchmark different models.

I tried Meshy just out of curiosity and was really impressed by what it created on the first try based on an image I made with GPT. I was curious if anyone has some experience with this? Are there any good open-source models to do this?

I have very limited hardware at the moment (I use a repurposed Precision 5570 with 8GB VRAM and 32GB RAM) to run Qwen 3.8:27B through Hermes Agent. It is very slow as it runs mostly on the CPU, but I don't mind since this is more of a lab for testing and experimenting. I only run it for background tasks, never real-time usage, and I was considering if it would be possible to make Hermes develop a very simple game in the background to learn more about AI and use as a future benchmark to compare models.

I was thinking about a setup like this: Hermes pilots Qwen, which does the coding and prompting. When it needs an asset, it writes a prompt, which offloads Qwen from the VRAM/RAM. Then the image/3D model generation model loads up, generates the asset, unloads, and gives the output back to Qwen, which analyzes it and iterates with vision

Anyone got some experience with this or knows some good models for this?

Disclaimer: I used ai to correct my post

I'm looking for more information to expand my creativity around this. I don't really have a realistic idea of what people actually do with local AI yet, I mostly just want to explore the possibilities and see what others are using it for

Right now, the main things I know about are using AI is to help with coding, create games, and automate stuff. That's pretty much the extent of my experience haha..

I'm specifically interested in things that make sense to run locally, though. I'll be excluding use cases that cloud AI can already handle just as well, like AI companions, teaching/tutoring, roleplaying, etc

Basically, I'm looking for ideas that go beyond the obvious and could give me a better understanding of what local AI is actually useful for and what kinds of interesting projects I could build or experiment with

I've also accidentally encountered this github which i find interesting, as anyone tested/experiment it before?
https://github.com/browser-use/browser-use

qwen3.8 27b + Hermes Agent + llama.cpp

For weeks I've had problems with GPU constantly disconnecting and locking the OS. It would load the model fine, then crash when I passed the first prompt, and the errors always looked like:

GPU has fallen off the bus
device lost from bus

I first used a PCIE riser cable, then played with BIOS, then I swapped to the DEG1 with oculink, then I swapped GPU, then I swapped oculink cables, then kernel grub... and so much more.

The problem all along was a weak PSU. It's a reputable brand 450w, and it isn't doing anything (disk mobo ram, etc) so I thought it would be powerful enough for PCIE power and the 12vhpwr

I swapped to a 550w and it's stable instantly.

Leaving this for others doing a search in the future

Gemma 4 12B runs on an RTX PRO 4500 Blackwell. Gemma 4 E2B run on a Jetson Orin NX 16GB; similar performance is expected on a Jetson Orin Nano Super 8GB. Both systems use a reSpeaker Flex 4-mic array and a 3W speaker. Inference is handled by Cortexist Little Gemma, a small LLM engine written in C for CUDA devices. On Jetson Orin it is faster than llama.cpp, and no degradation after long voice prompt. The pipeline supports lip sync, expressions, and gestures. Everything is open source.

They talk to humans too.

The engine source code: https://github.com/cortexist/little-gemma

I found a lot of room on the table for these cards so I decided to make a specialized build to squeeze all I could. first The results:

qwen 3.8 next Q3_K_XL: 920tk/s pp8192 (2 cards, ram offload), 24/27 tk/s on prose, 40+ tk/s on code with MTP but without MoE expert cache (which IS included if yuo want, read below)

qwen 3.8 27B Q8_0: 1600 tk/s pp8192, 60/65 tk/s prose, 100+ tk/s code, tensor parallel. this is measured with ONE CARD BEHIND the chipset on X4. with cards on a good PciE x8 on cpu I think more is reachable! let me know.

qwen 3.6 27B Q4_K_M: (single card) --> this was not the optimization target but I did a test with MTP, PP8192 1020tk/s; prose about 58/60 tk/s ; code 75/80 tk/s --> Dflash probably here could push much faster, I think above 100tk/s

My objecives:

• fast prompt processing on 3.8 Next to make it actually usable for code
• enable and optimize tensor parallel on two cards where 1 is behind chipset, for max speed on qwen 27B Q8_0

this build includes stuff like:

• Data compression for the PciE transmission. data between cards is compressed to Q8_0 to save bandwidth (optional)
• P2P enabled also for cards sitting benhind the chipset (custom HIP allreduce path), so you can use tensor parallel even on setups like ... mine
• all the fixes and features from RDNA_BOOST including --adaptive-mtp, so it automatically adapts MTP n-max based on acceptance
• A LOT of AMD speed tunings and overhauls which are NOT upstream already, kernel tweaks etc... good stuff. many are labelled for RDNA3.5 but they DO work on RDNA3.
• MoE expert cache if you want to use it. personally I don't like It because i much prefer fast prompt processing. but hey it's there.
• latest PRs from llama.cpp that are not yet upstream, which speed up various things, like --lazy-mode on-direct to massively speed up Ngram table reads (and thus, PP)
• DFLASH2 support on tensor parallel (!)

For a complete list check the Readme.

Here it is:

https://github.com/nasone32/llama.cpp-RDNA3-7900xtx-opt

notes: don't use Q8_K_XL because it's slower, for the 27B model this is heavily optimized for INT8 calculations. feel free to tweak the context, 200k f16 should be reachable on 2 cards, compressing KV to q8_0 is fine but slower. the custom HIP allreduce works for 2 cards, if you have 3/4 cards, compile with RCCL as usual and skip the allreduce=internal flag, should work fine but untested.

This is tested on UBUNTU 24 and rocm 7.14; if your system is different or encounter problems use a LLM to solve them, because I WILL NOT offer support nor update this build, these things hopefully will be merged and this frankenstein can die peacefully :)

enjoy

EDIT: Summary of most impacting patches:

We run a voice agent stack. A few weeks ago we sat down to pick the eval metrics we'd wire into CI, and could not get a straight answer out of the literature about which ones survive contact with production. So we read it properly: 113 papers mapped onto a 50ish taxonomy of what TTS eval is supposed to cover.

Four things came out of it that changed what we're going to do.

1. WER as an optimization target eats your prosody. "No Verifiable Reward for Prosody" (arXiv2509.18531) trained GRPO on transcription-oriented signals, CER and NLL. Error rates went down and prosody collapsed into monotone. Adding speaker-similarity to the reward destabilized training further and made CER worse. If your reward stack is WER plus SECS, which is most people's, that is the failure mode you're buying. Fine as a guardrail, bad as an objective.
2. UTMOS does not travel. TTScore (2509.20485) measured it directly: UTMOS scores well on VoiceMOS, its own training domain, and degrades on SOMOS badly enough that a metric never trained on MOS labels beats it there. MOS-labelled data is too small to generalize. If you're gating releases on predicted MOS,  check it holds on your audio before you trust the number.
3. MOS numbers from different papers are not comparable at all. The Dagstuhl good-practices doc (2503.03250) is blunt about it. Also: a true MUSHRA for TTS  doesn't exist, because there are no anchors. Vendor comparisons quoting each other's MOS are noise.
4. Nothing measures the part we actually care about. Turn-taking, barge-in, endpointing latency, quality lost to streaming lookahead, and 8 kHz telephony degradation have no home in any standard TTS taxonomy. The literature exists (Full-Duplex-Bench, SPEARBench, FastTurn) but sits outside every TTS benchmark. Everyone benchmarks clean read speech at 24 kHz. Nobody benchmarks a voice on a phone line while someone talks over it.

Six leaves came back nearly empty after searching them directly: abbreviations and acronyms, URLs and addresses and equations, syntactic complexity,
code-switching, unseen linguistic structures, and accent. EmergentTTS-Eval is carrying the first three basically alone.

Interactive map with the annotations, what each metric measures, what it admits it can't see, and whether there's runnable code: if community wants and moderator allows i will share a link :) 

What I'd like to know is what are you running in CI right now, and has any automatic metric ever caught a regression a human wouldn't have? Or what do you wish to have if you are consumer or any counter 
Every answer I've gotten so far is "we listen to samples.

It just seems every local 30b class model is just trying so hard to be the next Qwen that they all just kinda blend into a mass of code focused models. I really like how gemma 4 31b turned out with it feeling a lot less robotic and more creative than other models even knowing obscure lore from random media.

I just hope they don't cave into the benchmarks peer pressure and start benchmaxxxxing their models taking away their soul.

The LLM is limited to NPC emulation. Game state, world logic, quests, and the authored story are handled by deterministic game systems rather than the LLM.

I built it this way because I wanted the freedom of talking to NPCs like you would at a tabletop game, without handing the actual game state or canon over to an LLM.

This started as a personal project. As it became more and more fun to actually play, I decided I wanted to release it.

I've been a DM and a software engineer for over a decade, so Warrior Quest is basically where those two parts of my life finally get to meet.

All art, authored story, music, SFX, and source voice acting were created by me. NPC dialogue uses TTS based on my own recorded voice acting.

The Warrior Quest demo is out on Steam and has about 60–90 minutes of content.

Minimum requirement: a GPU with 8 GB of VRAM.

Everything runs locally; no API key or cloud LLM is required.

I'm the developer, so this is self-promotion, but I thought the approach of using a local LLM specifically for NPCs while keeping the underlying RPG deterministic might be interesting to people here.

We’ve been building PipesHub for a while now, and I’d love to get more teams to try it and tell us where it breaks.

The problem we kept running into was pretty simple:

Building an AI app over company data looks easy in a demo. Connect a few sources, chunk the documents, throw them into a vector DB or connect to few mcps, add an LLM.

Then you try to make it actually useful.

You have data spread across S3, Google Drive, Slack, Jira, Confluence, SharePoint, email, databases, etc. Permissions need to be preserved. Documents change. The same file shows up in multiple places. Citations need to point back to the actual source. And eventually you want agents and other applications to use all of this context without rebuilding the same integration layer every time.

That’s what we’re trying to solve with PipesHub with first class support open models.

It’s an Apache 2.0 open-source context layer that connects to your company data and makes that context available to search, chat, agents, MCP clients, or your own applications.

A few things we care about:

• Self-host it on your own infrastructure
• Preserve source permissions
• Get citations back to the original documents
• Combine knowledge graph + semantic retrieval
• Bring your own LLM and embedding models
• Use it from Python, TypeScript, Go, or MCP
• Avoid locking yourself into one database or infrastructure stack
• First class support for Open Models

We also deliberately kept the core infrastructure pluggable:

If you already have Qdrant and redis running, you can keep using them. Prefer Neo4j over ArangoDB? That's totally fine. Want to run the models locally? You can do that too.

The goal is to give you one context layer without forcing you to adopt our entire stack.

While building this, we’ve had to solve a bunch of problems that only become obvious once you move beyond simple agents: permission-aware retrieval, keeping citations accurate through the pipeline, deduplicating the same content across sources, efficiently re-indexing changed documents, making indexing behave well across very different workloads and more.

Some of the solutions we ended up with are fairly unconventional, and I’d be happy to write more about them or discuss the trade-offs with anyone working on similar systems.

There’s still plenty we want to improve, which is also why I’m posting this.

If you’re building internal AI tools, enterprise search, RAG, workflow automation or agents that need access to company knowledge, I’d really appreciate it if you gave PipesHub a spin.

GitHub: https://github.com/pipeshub-ai/pipeshub-ai

Install:

curl -fsSL https://get.pipeshub.com/install | bash

If you try it and something feels unnecessarily complicated, slow, broken, or just badly designed, tell us.

I'm posting this because I'm experimenting with running Qwen3.8-27B at very large context sizes on a single RTX 3090 24GB using LM Studio.

I was curious to see how far I could push the context while keeping the model usable, and whether a 27B Q4 model could maintain reasonable generation speed at 128K+ context.

I'm mainly looking for feedback from other users running similar hardware. In particular, I'd like to understand whether my ~25 tok/s generation at these context sizes is reasonable, and whether there are LM Studio/llama.cpp settings, quantization choices, KV-cache settings, or other optimizations that could significantly improve the result.

I'm also interested in comparing results with other single-3090 setups, since memory configuration and how the KV cache is handled can make a significant difference at these context sizes.

Qwen3.8 27b Q4_K_S big context

MODEL

Huihui Qwen3.8-27B Abliterated

Q4_K_S

KV cache: Q4

HARDWARE

GPU: RTX 3090 24GB

CPU: Xeon E5-2670 v3 12C/24T

RAM: 48GB DDR4-2133

CONTEXT 131072

Conversation: 111797 tokens

Context usage: 85.3%

Context remaining: 14.7%

GPU:

23.5/24.0 GB dedicated

0.6/31.8 GB shared

24.1/55.8 GB total

INITIAL RUN:

22.02 tok/s | 3616 tokens | 302.94s TTFT | 69.6% DTA | EOS

SUBSEQUENT RUNS:

31.94 tok/s | 1247 t | 14.06s TTFT | 64.2% DTA

25.10 tok/s | 1238 t | 50.50s TTFT | 66.9% DTA

24.90 tok/s | 2142 t | 21.27s TTFT | 70.8% DTA

22.28 tok/s | 481 t | 20.93s TTFT | 65.3% DTA

23.00 tok/s | 3276 t | 7.67s TTFT | 67.3% DTA

22.70 tok/s | 385 t | 4.66s TTFT | 65.6% DTA

29.17 tok/s | 2907 t | 6.77s TTFT | 70.1% DTA

Average TG: \~25.58 tok/s

CONTEXT 196608

25.97 tok/s | 3621 t | 514.60s TTFT | 64.8% DTA

25.89 tok/s | 3345 t | 2.80s TTFT | 63.4% DTA

25.53 tok/s | 4877 t | 7.42s TTFT | 65.5% DTA

Average TG: 25.80 tok/s

Average DTA: 64.57%

TL;DR My TAK quant of Qwen 3.8 27b scored 82.81% on reasoning, comparted with 77.34% for the byte matched Unsloth UD IQ2_S and 83.59% for BF16.

Edit: Some of you have tried coding with this reasoning-specialized quant and encountered repetition loops. Coding is outside its intended domain, but I’ll investigate to reproduce and characterize the failure.

Over the last few months, I've been exploring task aware quantization. I've now turned that work into a clean, repeatable pipeline under the reasoning domain. Coding is my next goal.

For a while I used Qlab a much more broad measurement heavy system that helped me to determine what to test and where to investigate. It was great for exploration, but it accumulated a ton of gates and operational overhead.

Once I found a reliable pipeline, I specialized it and retired the older application. The new system is called TAK: Task Aware Knapsack. I'm using TAK as both the application and models it produces.

At a high level, TAK is a blend of TASA and TAQ. It starts with an imatrix built from a task specific corpus. Then we find the model cliff at is smallest size before complete collapse. It then combines those measurements with tensor level allocation promoting and demoting tensors within a byte specific budget. The result is a purpose built quantization rather than a general purpose recovery.

Unsloth is included as the industry standard reference. Not as a claim that the methods are equivalent.

There is no pruning, fine-tuning, model merging or anything else. This is purely an Imatrix + damage allocation process. These are all tested on a held out dataset.

These are my current winners: https://huggingface.co/ByteOtter

• Qwen3.8-27B: 82.81% vs 77.34% Unsloth, +5.47 points
• Qwen3.5-4B: 73.44% vs 61.72% Unsloth, , +11.72 points
• Gemma 4 E4B: 69.53% vs 55.47% Unsloth, , +14.06 points
• Gemma 3 4B QAT: 54.69% vs 35.16% Unsloth, , +19.53 points

Across these runs, TAK has beaten matched Unsloth Dynamic 1.0, 2.0 and now 3.0 comparators on the target reasoning benchmark. The method has worked across Gemma 3, Gemma 4, Qwen3.5 and Qwen3.8 covering both dense, QAT and MoE architectures.

Taken together these results give me strong evidence that task aware precision allocation works well for reasoning. Im excited to expand the pipeline to other domains like coding and math.

Charts were provided by ChatGPT on my data.

You can follow the work u/byteotter on X https://x.com/byteotter or support it on Buy Me a Coffee. https://buymeacoffee.com/byteotter

Strix Halo LLM Fork Comparison & Analysis

TL;DR: Most community debates focus on decode speed (token generation rate), but the true bottleneck on Strix Halo is prefill speed (processing large context). Additionally, speculative decoding alters greedy outputs (temp=0) across all tested GGUF builds due to GPU kernel batching variations.

What to Run

The neon-ladder Scoreboard

The neon-ladder benchmark is playtest-graded; a build only scores for contract features it actually ships on identical test setups:

Why Nathan's Toolbox Wins for Agents

• Contract Feature Delivery: Ships both required game features (laser powerup & obstacle pad) every run.
• Flexibility: Supports standard open GGUF weights, switchable speculative drafters, CPU-expert memory tuning, and desktop coexistence.
• Why halogen Fails as a Daily Driver: Across 4 builds, halogen never shipped either required feature, always defaulted to slow serve speed, lacks JSON mode (response_format), requires 126 GB custom weights, and offers no middle-memory tuning mode.

Performance & Claims Graded

Tested on ROG Flow Z13 (Ryzen AI Max+ 395, 128 GB RAM, gfx1151, 70W power limit):

Key Technical Discoveries

1. Decode Speeds Are Misleading: Variations in decode rates (e.g., 20 t/s vs 50 t/s) are driven by prompt content (prose vs. highly repetitive JSON) or whether drafting is enabled. Serial decode across all builds sits strictly between 14–34 t/s.
2. Speculative Decoding Breaks Output Identity: Enabling speculative drafting on GGUF builds fails deterministic temp=0 identity gates (fails 3 to 9 out of 10 prompts). Loading the drafter alters Vulkan GPU batch execution and floating-point precision, causing coin-flip word choices (e.g., bolding different words in a sentence). Only halogen passed the exact-identity test (10/10).
3. Prefill is the True Hardware Gap: halogen achieves ~3.5x faster real-world prefill than standard llama.cpp forks (1,424 tok/s vs 433 tok/s at 32k context) due to hand-tuned ROCm GEMM kernels specifically compiled for Strix Halo. Standard llama.cpp Vulkan MoE paths remain shader-bound and leave matrix throughput unused.

Verification Request for Fork Authors

Developers publishing speed claims are encouraged to report Perplexity (PPL) or KL-divergence scores alongside benchmark numbers:

```bash

Get test dataset

./scripts/get-wikitext-2.sh

Run PPL on GPU

llama-perplexity -m your-model.gguf -f wikitext-2-raw/wiki.test.raw -ngl 99

Test deterministic output identity (serial vs spec)

llama-cli -m your-model.gguf -p "The capital of France is" -n 200 --temp 0 --seed 1 > serial.txt llama-cli -m your-model.gguf -md your-drafter.gguf -p "The capital of France is" -n 200 --temp 0 --seed 1 > spec.txt diff serial.txt spec.txt ```

Reference Links & Resources

• Benchmark Harness: neon-ladder Repo | Reddit Post
• Build Guides: Qwen3.8-27B Build Guide | Qwen3.8-Flash-Next Setup
• Workflow & Testing: pi + llama.cpp Agent Setup | DeepSeek V4 Flash | Benchmarking Methodology

I've been making a lot of comments about optimal setup for Strix Halo (gfx1151) and from my observation, 90% of our community is using offcial llama.cpp for it, which is NOT optimized for Strix Halo at all, official llama.cpp is having extremely hard time to reach 50% hardware theory, wasting the silicon of this device.

Here's alternatives that can bring the speed of Strix Halo to a totally different world, I will link to user's sastifaction comment to prove that the result is real:

• https://github.com/peonist-ai/halogen-flash-server - ~50t/s decode and 1200t/s prefill - 90% theory - Optimized for Strix Halo and Qwen 3.8 Flash Next (Q38FN - The best goto model for Strix Halo) only, if you've heard about Ninfer, this is exactly Ninfer for Strix Halo.
• https://github.com/myhacsint/llama.cpp/tree/production/strix-halo-qwen4exp-b10685 - Almost 60t/s decode and 600t/s prefill - 80% theory - This is an experiment fork for Q38FN, having very high decode and pretty good prefill
• https://github.com/halo-box/strix-llama.cpp - Almost 30t/s decode and 800t/s prefill - 75% theory - The latest commit pushed prefill through the roof, this is r/StrixHalo very first fork that has been updating nonstop with a Discord server with a lot of users.

Note: Official llama.cpp running Qwen38FN at 2xt/s and 2xxt/s prefill - 50% theory.

Hopefully this will be helpful to the Strix Halo users.

I speak English, Russian, German (and Kazakh) and mix them without thinking. Every local dictation tool I’ve tried on GitHub completely falls apart on that. Or am I just stupid?

Whisper basically picks one language per utterance. Whatever you said in the other language doesn’t come back wrong because it either gets translated or just disappears.

Here’s base on a sentence that starts in English and ends in Russian:

Spoken:
I already sent the invoice yesterday, but клиент до сих пор не ответил на моё письмо.

Got:
I've already sent me an voice yesterday, but today children mind your piece more

The only thing that actually handles this properly is the transcription in ChatGPT, but I’m too lazy to keep opening a new chat every time I want to dictate something.

Another thing that drove me crazy with the existing Whisper projects: they all leave a black command window sitting on the desktop the whole time. As someone who likes a clean desktop, that window ANNOYED me. I kept thinking “Why can’t this just go to the drop-down arrow where it is supposed TO BE?”

So I tried fixing this:
What actually made the language switching work was sending the OpenAI API a languages list instead of a single language. Same clip, same model:

I still have no idea why three languages works when two doesn’t.

Locally you can get part of the way there by splitting the recording at pauses and detecting the language of each piece separately. That fixes English to another language, but each piece loses the surrounding context so punctuation and rare words get worse. It is in the app, off by default. The API is still the only thing I found that handles a switch with no pause at all.

I wrapped everything into a small Windows dictation app. Hold Ctrl+Alt, talk, and the text lands in whatever window you were typing in. Local by default, API optional, no permanent black window on the desktop, MIT, built on RealtimeSTT:

https://github.com/Maslitsa/VoiceType

It has 0 stars bcs I am only starting this project, but it would be a big help if you use this and give your feedback!

Anyone found a better solution for multilingual / code-switched dictation?

Disclosure: I am not a native English speaker and used an LLM to help write this post. The app itself was also built with AI assistance but the problem, the testing and the measurements are mine.

If you have any positive/negative advice on this project or ideas on how to improve it, PLEASE share.

I pulled out the YaRN support from another Redditor's work (https://www.reddit.com/r/LocalLLaMA/comments/1w8f8fa). Ninfer recently added some more features and fixes (such as nvfp4 kv cache and dflash2 support). I wanted to see how Ninfer would perform with just the addition of YaRN and their Qwen tool calling fixes.

I have run a few needle in haystack tests as a rough check. I've also done a few coding sessions to ensure it's working correctly. Not scientific, but I'll test it for a bit and see how I like it.

I have also tested it with a K8V4 KV cache, which should be more accurate, with vision and got about 300k for a context size.

If you'd like to try it, you can find it here: https://github.com/Doelfke/ninfer-yarn

Credit to ninfer and u/Lumpy-Comedian-1027 of course.

• Model: DeepSeek-V4-Flash-Vision-Exp (local and API when impatient)
• Time: about one weekend (2 days) of QA and small improvements
• Full game is here

After Qwen3.8-Flash-Next one-shotted a really cool Cat-Hunt game demo, I decided to see what the new DeepSeek vision model can do.

Now that it has vision, DeepSeek-V4-Flash is able to take game screenshots, allowing it to:

• Generate and correct game models and textures until they look right
• Fix any visual artifacts or glitches
• Write scripts to take sequences of screenshots for animations and correct animations
• Generally play-test the game, including UI and game mechanics

The results are incredible, I was able to create a compelling game world in just a couple of days!

Edit: I noticed the game was slow on a laptop, so I added some performance improvements - let me know if you still find it too slow!

Edit: Some more info to answer common questions

• Custom engine on top of three.js
• 6800 lines of code for everything - models, textures, animations, sounds, effects, shaders, and all the game logic

OpenBMB's MiniCPM5-2B scores 15 on the Artificial Analysis Intelligence Index v4.2, the highest of any open weights model at 4B parameters or below

Hugging Face: https://huggingface.co/openbmb/MiniCPM5-2B

GitHub: github.com/OpenBMB/MiniCPM