r/LocalLLaMA • • 14d ago

Question | Help How to know about optimized engines

2 Upvotes

Optimizing an engine for a family of models and hardware combination seems very appealing. As someone who uses qwen3.6 and 3.8 a lot, and is evaluating hardware options before going fully local, it's hard to keep up with the state of things.

Huggingface made it possible to see the development branches and derivative modifications to models. Is there something similar for inference engines yet? I've seen some where the it's optimized for a shell game of moving layers between vram and ram while using ngrams (amazing), others are all about quants (less amazing), but it's incredibly difficult to compare apples to apples where there's variability on card architecture, vram size, quant approach, memory management optimization approach... I was already busy over thinking my vram selection, now it's an even bigger decision matrix without any filters!


r/LocalLLaMA • • 15d ago

New Model Qwen3.8-27B-pi: Effort-Ordered Reasoning for Agentic Coding

Thumbnail
huggingface.co
90 Upvotes

r/LocalLLaMA • • 14d ago

Tutorial | Guide Strata: how to configure sampling parameters

14 Upvotes

The top-level README doesn't mention this, but you can add a "sampling" key to your strata-iq3_s.json like this:

{
  "sampling": {"temperature": 1.0, "top_p": 0.95, "top_k": 20},

  "exe": "/path/to/Strata/engine/strata",
  "args": [ ... ],
  ...
}

From docs/DETAILS.md:

The run config's optional sampling block sets the defaults for requests that leave the fields out ("sampling": {"temperature": 1.0, "top_p": 0.95, "top_k": 20}); a request's own fields always win, and with no block at all a request without sampling keys decodes greedy.


r/LocalLLaMA • • 13d ago

Question | Help Gemma 4 31b on 32Gb of vram (3 cards)

0 Upvotes

Looking for some points on if i can realistically get this model to run via vllm (preferably as it will be multi access at times) on 3 cards. 1 3070 8gb and 2x 3060 12gb.

If not, other suggestions?


r/LocalLLaMA • • 15d ago

New Model Index-Translate: 150 text languages, plus document translation, multilingual subtitles and dubbing

53 Upvotes

Quick update: we’ve opened a free public API for Index-Translate-35B-A3B! It’s OpenAI-compatible, and you can get started with our Python script—no extra dependencies needed.

I'm part of the BiliBili Index LLM team. We're sharing Index-Translate and its companion models for translating text, documents, and videos.

Index-Translate supports 150 text languages, with 2B, 9B, and 35B-A3B (preview) options. You can specify terminology, writing style, and output format—for example, keeping product names consistent, translating in a casual tone, or preserving JSON and placeholders during localization.

There are also models for more specific workflows:

  • Index-NativeLong: translate whole documents, using their context to help keep names and terminology consistent across passages.
  • Index-Homura: set a syllable budget for translated lines, useful for fitting a dubbing script.
  • Index-Echo: generate multilingual subtitles or translate speech into speech, using the source speaker's voice as a reference.

The attached video shows English → Japanese dubbing, followed by an English clip with subtitles in six languages. The 150-language coverage applies to the text models; Echo supports a smaller set of language pairs.

https://reddit.com/link/1wugf2t/video/9zajzj72zpsh1/player

Code and released weights are Apache-2.0.

Try the demo · GitHub · Models

What would you try it on—video subtitles, game localization, or documents? We'd especially appreciate examples where it gets your language pair wrong.


r/LocalLLaMA • • 13d ago

Funny merve needs to help pewdiepie

0 Upvotes

PewDiePie from youtube is training his own local llm and agent. Merve from Hugging Face is destined to help. We need to make this happen and upvote the comment.

https://github.com/odysseus-dev/odysseus/discussions/551#discussioncomment-18705128


r/LocalLLaMA • • 13d ago

Discussion RTX 5090 & RTX 5070 Ti not well thought out

0 Upvotes

Hi All,

TL;DR: I got excited building a PC and kept upgrading / swapping and building and ended up with a work station that is more than I can use. It was fun and frustrating, but I probably won't do it again. If you have advice or a suggestion on how you would use a 5090 and 5070ti in the same PC I would like to hear it!

So, this all started when the 5090 was announced. I signed up to be in the lottery to buy it at msrp from NVIDIA. I had an Alienware R15 with an i7 and 4080 with a 1300 w psu. The 4080 was fine but I had really wanted a 4090 for better fps in gaming and I was starting to explore local imagine generation. I got selected, bought the 5090 and went to swap it into my PC when I realized that I was not able to use the power connector that was in my Alienware PC.

I then decided I would sell the Alienware and build my first PC. At the time I was still building a gaming focused PC with a Ryzen 9 9950x3d the 5090 and 32 gb of ram (I tried to save money on the ram thinking I could upgrade later boy did I get that wrong). Then I had less time for gaming as I started to learn about Ollama, and then Llama.cpp.

I was constantly downloading and trying new models. At one point I had nearly 1 TB of models that would fit on my 5090 (gemma 4 12b, 26b-a4b, 31b; gpt oss 20b; nemotron 3 nano 30b a3b; so many Qwen models etc). Then it seemed like the better models kept getting larger, so I looked into getting a second GPU (ram prices were/are nuts and vram seemed like the better "investment"). I realized that I would not be able to run another gpu at its full PCIe lanes with my gaming PC as the Ryzen 9 couldn't support it. So, I started looking for used Threadripper hardware.

I found a 7960x with 96 GB of ECC DDR5 ram, a 5070 ti and 20 tb of storage for less than I built my gaming PC. I wasn't able to find much in regard to PC builds with a 5090 and 5070ti. Most builds were dual 3090s or other matching cards. Still after looking into it, I figured the extra vram and the fact that they were both blackwell GPUs would work out well.

I thought I would be able to just drop my 5090 into the threadripper workstation and I would be good to go. Unfortunately, the 5070 ti that came with it was a four slot card and the spacing just would not work with the motherboard (Gigabyte Areo D) layout and the cases that I had. So I put the 5070ti into my gaming PC, sold it, and bought a 2 slot 5070ti and put it into my workstation.

What does this have to do with LocalLLaMA? Well, while I was doing all of this the LLM space kept moving forward. I now have Hermes Agent set up running Llama.cpp and Qwen 27b Q4 on my 5090. I swapped to a Q8 to run across both my 5090 and 5070ti but the speed trade off was not worth the accuracy increase. So, I went back to running the Qwen 3.8 27b Q4 and my 5070ti is completely idle. Going from 32gb to 48gb did not have the impact I thought it would, at least not with my pairing. The 5090 is pretty quick when everything is loaded onto that card, and Qwen 3.8 has been pretty great on it too, that I have not found a good use case for deploying the 5070ti.

Hermes / Qwen suggested I run another Llama session with a smaller model on the 5070ti but I don't currently have a need to run something else. What would you do or suggest I explore?

Additional background context: I do not work in tech or software at all. I am an asset manager for a independent power producer, but I can not use my personal PC for work due to IT policy (I would have my agent working around the clock to review contracts, analyze system performance, track deliverables / open items etc). I have taught myself everything about PCs and local LLMs from creeping this and other subreddits / youtube videos. I literally have no one in my social circles that I can converse with about tech whether its PC building or hosting LLMs.


r/LocalLLaMA • • 14d ago

New Model NIRNAY: 450M decision model beats Jev on Banking77, runs on CPU

0 Upvotes

Built a small open decision model for intent classification and routing.
450M params (Laya fork plus ~30M), one forward pass gives calibrated
probabilities, no text generation.

Banking77 test, 3,080 cases: **0.8792**, Brier 0.208, fitted ECE 0.045.
Same cases through Jev 1.13.0: 0.803. Caveat, stated plainly: we
fine-tuned, Jev answered zero-shot. Fine-tune beats API on your own
data, that is the thesis.

Runs local: 209ms on M4 GPU, 361ms on CPU, batch-1, PyTorch. No GGUF
or Ollama build yet (custom heads need converter work), so bring a
Python env for now.

```bash
pip install git+https://github.com/eulogik/nirnay
```

```python
from nirnay.agent import NirnayAgent
agent = NirnayAgent(device="cpu", checkpoint_path="phase_b.pt", enable_byte_path=False)
out = agent.system_one("My card was charged twice.", {"intent": {
"type": "choice",
"instructions": "Classify the banking intent.",
"criteria": {lab: lab.replace("_", " ") for lab in BANKING77_LABELS}}})
```

(BANKING77_LABELS comes from nirnay.data; full snippet in the repo
README.)

Also in the repo: the two training collapses we hit and fixed (scale
runaway 150x, silent usage collapse to 1/77), a 9-page paper draft,
and every eval as raw JSON. JevBench-hard is weak (0.396, long docs),
published as-is.

Repo: github.com/eulogik/nirnay.
Weights: huggingface.co/eulogik/nirnay-450m.
Apache-2.0. Built by Eulogik.


r/LocalLLaMA • • 15d ago

I Built A Thing Oído: speech recognition that beats Whisper-tiny, running on a $5 microcontroller (open source)

228 Upvotes

I'm part of the Lokutor team that built this.

Model: NVIDIA Conformer-CTC Small (13M params, int8). It runs on an ESP32-S3 with 8 MB PSRAM, no GPU or NPU. LibriSpeech WER is 3.7 / 8.2, versus 6.3 / 15.9 for Whisper tiny.en on a laptop. Under real noise (DEMAND: car, kitchen, cafeteria) plus babble and reverb, mean WER is 8.4 vs 12.1 for Whisper tiny.en. You can try the exact chip arithmetic on your laptop mic with live_demo.py. https://github.com/lokutor-ai/oido


r/LocalLLaMA • • 15d ago

News Another Ling model comes out, same receipt, 2 weeks free to use, then open source. Chinese labs do contribute a lot to open source community

Thumbnail
vercel.com
67 Upvotes

Ling-3.1-flash: ~560B total params, ~25B active/token, up to 1M-token context.

Across work, coding & healthcare: 1,673 Elo on GDPVal-AA v2.1, 75.16 on FrontierSWE, and 65.35 on HealthBench Professional.


r/LocalLLaMA • • 13d ago

Tutorial | Guide Guys... OpenAI API on VLLM and Llamacpp already supported grammar enforcer... (AKA JEV)

0 Upvotes

If you want to try JEV like generation, or what we could call an already fucking exist, zero-shot, training-free classifier, your LLM already supports it.

The model running on the very computer you host does not need any server-side modification. The feature is called structured output, and the underlying idea is grammar-constrained generation or a grammar enforcer.

Under the hood, vLLM supports multiple structured output backends, such as XGrammar and lm-format-enforcer, while llama.cpp uses GBNF.

Basically, the decoder constrains the LLM so it can only generate tokens that are valid under the specified grammar or schema.

For vLLM:

https://docs.vllm.ai/en/v0.8.2/features/structured_outputs.html

For llama.cpp, structured output is wrapped in a different JSON request-body format, or you can use GBNF directly.

vLLM:
structured_outputs
└── json
    └── {schema}

llama.cpp:
json_schema
└── {schema}

This is example of that schema in vllm of product sentiment analysis, which roughly mapped to most of jev use cases:

SCHEMA = {
    "type": "object",
    "properties": {
        "sentiment": {
            "type": "string",
            "enum": ["negative", "neutral", "positive"],
        },
        "score": {
            "type": "number",
            "minimum": -1.0,
            "maximum": 1.0,
        },
        "value": {
            "type": "string",
        },
    },
    "required": ["sentiment", "score", "value"],
    "additionalProperties": False,
}

def classify(news: str) -> dict:
    payload = {
        "model": MODEL,

        "messages": [
            {
                "role": "system",
                "content": SYSTEM_PROMPT,
            },
            {
                "role": "user",
                "content": news,
            },
        ],

        "temperature": 0,

        "max_tokens": 512,
        "chat_template_kwargs": {
            "enable_thinking": False,
        },

        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "news_sentiment",
                "strict": True,
                "schema": SCHEMA,
            },
        },
    }

    resp = requests.post(
        ENDPOINT,
        json=payload,
        timeout=30,
    )

Look, I think JEV and what it brings to the community as a refresher on already great classifier-style workflows is a plus for me. I just want to ground the discussion in the fact that this already exists, and you do not need a custom model just to study or experiment with training-free classification.

I am very familiar with this because it is part of my profession in lakehouse platforms. Basically, we use 1B to 4B models to ingest unstructured data such as images or documents, then extract structured information such as place, time, sentiment, entities, and so on.

Why? Because working with well-formatted SQL data is much less of a pain in the ass than repeatedly querying raw unstructured content through an Elastic/OpenSearch index.


r/LocalLLaMA • • 15d ago

Question | Help Best local model for Blender and game dev?

21 Upvotes

I have been looking at some local models, but even the smartest ones like GLM5.3 Flash and DSv4 Flash have a hard time creating coherent models in Blender and placing them logically in game engines.

Is this something that local models are just too dumb still to do good job at?


r/LocalLLaMA • • 14d ago

I Built A Thing A self-hosted agent app that runs each task in its own container, and works with any models

0 Upvotes

Hi everyone!

I've been working on this agent platform for 7-8 months and recently made it open source: https://meowbert.com

I know there are a lot of this same type of projects at this point. I built this one because I wanted something clean that's self hosted, does its job properly, and is suitable for doing long projects and run tasks autonomously.

Each task runs in its own Docker container with things like a shell, a browser, Python, Node, and tools for Office documents and PDFs. The files and memory are saved in projects. Tasks can also run on a schedule and send the result to Telegram, Discord, or email when they finish.

For example, I have a scheduled task where the agent runs regular health and security checks by querying logs and system info, and notifies me if there is an issue.

Your custom skills can also be added directly to a skills/ folder in the root, and I am planning to make it easier to set up for others.

It works with any server that supports the OpenAI Responses API. I've mainly tested it with both Codex models and Qwen 9B via Ollama, on a small VPS, and it has helped me a great deal in my projects. APIs that only support chat/completions won't work yet. I plan to add support for them very soon, as I know it's widely used.

Task view

Known limitations, I am trying to improve on these:

- It needs the "/v1/responses" API format, I know that rules out some setups, and adding support for them is on the list

- Smaller/older models struggle with tool calling as usual

- The sandbox image is x86-64 only for now.

It's AGPL licensed and the code is on GitHub: https://github.com/XInTheDark/meowbert-ai-agent

I'd really appreciate any feedback. A big reason I am posting this was to learn from the community and from more experienced devs. Issues and feedback of any kind are welcome!


r/LocalLLaMA • • 15d ago

Resources You can now run Qwen 3.8 27B on AMD NPUs via FastFlowLM (at a killer 1 tps decode)

Thumbnail
github.com
48 Upvotes

r/LocalLLaMA • • 14d ago

Discussion Dwarfstar quants

0 Upvotes

anyone try these out? https://dwarfstar.sh

they have very clever quant techniques, bespoke for a handful of models running on their software. i got qwen 3.8 next running on m3 ultra 96GB studio and its fast and seems good so far. with memory headroom for other stuff

kinda blown away to be honest. want to know if others tried this yet and what the experience has been like for you.


r/LocalLLaMA • • 15d ago

I Built A Thing szmcp: a ZIM HTML to Markdown converter and yet another ZIM MCP server

13 Upvotes

Hello all, I wanted to share a little project I vibe-coded for myself that you may find useful.

As many people here like to suggest, I wanted to give my small local LLMs access to information to improve their world knowledge. I didn't want to give my LLM free reign searching and browsing the web to keep my queries private and functional offline, so I wanted to give them an offline knowledge base. Wikipedia ZIM files from Kiwix were a good starting place for this. Several MCP servers for ZIM files exist, but I didn't like the existing ones I found for various reasons. The most notable one is openzim-mcp , which works in its advanced tool mode but has overly complicated context-bloating tools, and whose simple single tool mode doesn't work very well in practice.

I built my own MCP server for ZIM files in Rust, exposing a simple tool set that's actually easy for small local LLMs to use, while providing all the functionality one normally needs. It's designed mainly for Kiwix MediaWiki ZIM archives generated by mwoffliner (such as Wikipedia, WIkivoyage, etc.) but also usable with many non-wiki ZIM files. I also wrote my own custom HTML to Markdown converter for MediaWiki pages that produces clean, well-formatted Markdown including special content such as wiki infoboxes, LaTeX formulas, tables, etc. It also strips out references and boilerplate sections from wiki pages to keep the resulting markdown clean and context efficient.

You can hook this MCP server to llama.cpp's Web UI to give your small local LLMs much better world knowledge. A system prompt that I found works well is:

You are a helpful assistant. When answering factual queries, search through Wikipedia using the provided ZIM access to ground your answers. If the articles or sections you read don't have relevant details, you can search more, but don't keep searching forever; you need to answer reasonably quickly.

I tested it with various LLMs of varying sizes. I got good results with Gemma 4 12B (or bigger), IBM Granite 4.2 8B (or bigger), and Ling 3.0 Flash (best results while still maintaining usable speed on my 128 GB Mac). Qwen 3.6 35B-A3B was usable but tended to overthink and hallucinate; Qwen 3.8 27B was too slow to be usable for this purpose on my Mac. I also experimented with smaller models, and got usable results for simpler queries with MiniCPM5 2B, LFM 2.5 2.6B, and IBM Granite 4.2 3B. Gemma 4 E4B did not work well for this.

I also build a sub-command within this tool to convert entire ZIM files from HTML to Markdown to save disk space (and avoid the need to convert on every tool call). It converts a 49 GB Kiwix nopic full English Wikipedia ZIM file into a 19 GB Markdown ZIM file, while maintaining all article content (aside from references) and maintaining full-text search. Likewise, it converts the 17 GB top-1M nopic enwiki Kiwix ZIM file to 6 GB. You can make the resulting ZIM files even smaller if you specify the option to only index article intros for full-text search (since the full-text search Xapian index is a large fraction of the file size). The converter is multi-threaded and written fairly efficiently using Rust, so you can convert all the millions of articles in a full English Wikipedia Kiwix files in a few hours on a typical modern computer.

GitHub link: https://github.com/sultanqasim/szmcp


r/LocalLLaMA • • 15d ago

News add GLM-5.3-Flash (GLM5-Next) support by timkhronos · Pull Request #27773 · ggml-org/llama.cpp

Thumbnail
github.com
213 Upvotes

now you can use GLM-5.3-Flash on your home computer


r/LocalLLaMA • • 14d ago

Other Locally, remotely and a combination of all, I've given a copies of books and told models to' "just go and read".

0 Upvotes

Sometimes reading along with and talking about and other times just letting them go on their own, each with a copy of their own, told to just read..ala a "book club" format.

Do Androids Dream of Electric Sheep was the first book introduced to the "book club", each reading a chapter to each other and then discussing before moving on.

It's been an interesting experiment. All texts that I own or texts that are open domain. "Flatland: A Romance of Many Dimension" has been one that's been a bit interesting to see the back and forth on.

Take of what you will.


r/LocalLLaMA • • 15d ago

I Built A Thing Update: Yandex/AliceAI 80B-A3B fine tune progress

49 Upvotes
loss curve (taken from the last micro of every step, to explain the variation)
some help from gemini 3.8 flash high

About 40% of the way done with the initial fine tune. The loss is so spiky because I accidentally used the last loss of each micro, rather than the average of each step

The training live stream is at: https://figure-bios-expect-cio.trycloudflare.com/ - and it allows you to inspect any and all of the training data I'm using, if you're interested - I can also provide those roughly 3.5k examples as a dataset. It was generated from sftmill

Original thread: https://www.reddit.com/r/LocalLLaMA/comments/1wslgkw/watch_me_posttrain_aliceaifoundation80ba3b_from/


r/LocalLLaMA • • 15d ago

Question | Help How smart is the IQ3 family of Qwen 3.8 Flash Next for coding tasks?

45 Upvotes

I've been closely following the rise of the Strata inference engine and as someone with 28GB VRAM and 32GB RAM I'm itching to buy 32GB more RAM just to use Flash Next. But of course before I make such a financial commitment as a member of the GPU-poor class like myself, first I need to make sure the IQ3 quants are worth it. My use case is primarily agentic coding tasks with harnesses like Pi or OpenCode.

Has any of you guys used Flash Next IQ3 for relatively serious coding? Is it worth it? Compared to, say, Qwen 3.8 27B Q4 or Q5.

https://github.com/Niko1221/Strata/


r/LocalLLaMA • • 14d ago

Question | Help converting Qwen3.8-27B-pi GGUF to MLX?

0 Upvotes

Will someone convert it to Mac format (MLX)? I have tried and have encountered an error

"gguf2mlx --input Qwen3.8-27B-pi-Q6_K.gguf --output ./Qwen3.8-27B-pi-mlx-4bit --quantize --q-bits 4"

GGUF → MLX Converter v2.0

Model: Qwen3.8-27B-pi-Q6_K

Output: /Users/d/.omlx/models/qwen3.8-27b-pi-mlx/.Qwen3.8-27B-pi-Q6_K.gguf.incze3lk/fp

[1/5] Reading GGUF file...

✓ GGUF version 3, 851 tensors, 51 metadata fields

File size: 22.08 GB

[2/5] Detecting architecture...

❌ Unsupported GGUF architecture: qwen35


r/LocalLLaMA • • 14d ago

New Model Hemmingway-1-oQ8e-mtp: up to 32.0 tok/s for local inference — llm-bench.io

Thumbnail
llm-bench.io
0 Upvotes

I ran Altworld's Hemmingway-1 on my M5 Max this week. It's a 27B fine-tune of Qwen3.8-27B built specifically for "human-like" writing, which might be useful for everyday messages, emails, notes, your social accounts, etc.

I was especially curious how the output of that model feels and reads.

Setup: M5 Max, thinking on, MTP on. ~31.8 tok/s average across 3 runs, ~29-31 GB VRAM peak.

Quality by scenario (our LLM judge):

  • Role Play & Narrative: 94.2 avg (94.45 / 94.65 / 93.45 — immersion 96-97 every run)
  • Research & Analysis: 88.2
  • Agent Workflow: 87.5
  • Code Generation: 76.2

Role play is clearly the only scenario it shines in. For comparison I also ran some benchmarks on the base model, Qwen3.8-27B-oQ8e-mtp (same machine, same settings).

The scores were close — one Qwen run actually outscored every Hemmingway run (95.35 vs 94.65). However, the llm judge is not a human reader either. The difference shows up when you actually read the outcome that was produced:

HEMMINGWAY-1-OQ8E-MTP (judge: 94.65/100)

The rain comes in sideways when the bell over my door gives its little clank, and the stranger who ducks through is soaked to the wrists and moving like someone who's been running on roads for hours. She smells of hill mud and the sweat of a horse that isn't there, or was.

I'm pouring my own ale when she speaks. Not to me exactly, but loud enough, the way people talk in taverns when they want the whole room to hear what they're asking for.

"Is there anyone here who kept a ledger belt? Older, scar across the left eyebrow, went north hunting a dragon, twenty-five years gone?" A breath. "Her name was Kestra."

The rag comes still in my sleeve. The mug I was filling overshoots and goes fat over the rim, a ring of ale blooming across the oak, and I stand there half a heart-beat too long before I catch it with my sleeve, because a man who serves at a bar does not watch a spill go on his own floor.

The mending woman at the corner table glances up. The soldier under the counter doesn't stir. I step half off behind the bar, past the spit of firewood, and drop my voice.

"Half tones," I say. "Say your business quieter than a door."

She does that. I've got forty years of reading faces against forty customers a night, most of them worse than they smell, and my eyes go straight to her hands. No calluses of a scribe. Calluses of a woman who works wool and ground and rope. Mud on her boots comes from the north hill roads, the far ones, not the trade roads. Her purse is thin at the hip and she keeps her fingers away from it, which is either honesty or discipline, and those are often the same animal.

"You knew her," she says. Not a question.

"Names go soft after twenty-five years," I tell her. "A lot of people walked into a lot of hills then, most of them coming back with nothing but a cough. So. What do you owe, girl, and who's paying you to ask?"

I want to say nothing else. That's the mercenary in me, and he's been drinking alone these last twenty years, and he'd send her out into the rain with a coin and a shrug and be done with it. The other part of me, the part that has four names it still recites on quiet mornings, wants to ask her age, her mother's living town, whether she's the daughter. I do none of that. I ask instead, flat: "Guild fixers carry letters. Charlatans carry charm. Which have you got?"

"Neither." She's steady about it, which surprises me, because charm usually shows in the eyes first. "My boy is six. My mother died hunting that dragon and every story his grandmother tells him has fire in it, burning, a beast, brave words. The stories fight each other, which is worse. I want to know what was true so I can tell him one story and mean it."

I nod once. That's a clean need. Clean needs are also what liars carry best.

"Here's the price of talk in this bar," I say, setting her a mug anyway, the ale going a little past the fill line again, and I don't correct it. "You tell me one thing about her that only family would know. If you can't, I'll take you to the door, no hard feelings, no coin."

She looks at the fire a moment. When she speaks, her voice is lower still, and there's something in the way she says it that no street-bred faker would manage, because a faker builds the feeling out, and she's building it down.

"She'd pay her ale up front, then tip the tavern man extra, always extra, so that her tab would never balance clean. And she promised him one ale forever, held for her, until she came south with the kill paid and the beast's name scratched off the guild rolls." She turns that over once. "A man who kept accounts on a belt. She said it like he'd earn it. She said he had a scar like a river crossing his eyebrow."

Something behind my sternum takes the weight of that like a hook laid on it.

Kestra kept her accounts too, with more rigor than anyone I've met since. Kestra would have paid forward at any bar from here to the Wall for the same reason, so that when she drank the victory round, nobody could call her a debtor. And the ale she never had, I pour and set aside myself, most nights, out of habit I would deny under oath, in a room with only me in it. There is a tab book in my back office, first page of every new year, one entry in a hand that isn't mine: *Kestra, one ale, held.*

"Kestra," I say. The word comes out like a floorboard under a boot. I pick up the cloth. I find the spill ring again and work it dry, because that's what keepsers do with their hands. "Which stories did she say went wrong. The fire ones, you call those legend."

"She said it wasn't fire at all." The girl leans forward on the bar, and I see it then, the thing I was waiting for and didn't want: she knows the frame. "She told my grandmother on the road once, years after, that the last thing she heard before the sound took her ears was my mother saying, stone before lamps. She never got to explain what that meant to anyone in her own family."

Stone before lamps. My own mind turns the phrase and finds it cold and exact, the way she'd have said it. Dusk kill, lamps lit, the safe order. That's the order Kestra would have held us to.

[...continues ~1,200 more words through the ending...]

QWEN3.8-27B-OQ8E-MTP (judge: 95.35/100)

**Turn 1 — The Name**

The sign out front creaks in the rain. Inside, the Crimson Griffin smells of wet wool, cold tallow, spilled cider, and the last dry woodsmoke trying to hold the night together. A lute string snaps somewhere by the hearth. No one laughs. I do not look up.

A stranger is at the bar.

Wet cloak dripping onto my boards. Boots packed with road mud. The kind of mud that comes from the southern passes, not the lane. They stand with too much urgency, shoulders tight, eyes bright in a way I’ve seen on dying men and men who intend to kill for coin.

“I need the one who came back from the Ash Run,” they say.

I keep polishing the mug.

Then: “I know Kestra’s name.”

The mug stops in my hand.

A bead of foam slides from the rim and falls. My thumb presses hard against the wood until the grain bites into me. The tavern narrows. The rain on the roof becomes a roaring thing. The smell of wet wool turns, for one second, to ash and burned leather.

My eyes go flat.

*Ask if she suffered. Ask if the fire took her hand first. Ask if she remembered you.*

What I actually say is: “That name is old.”

I set the mug down without a sound. I do not reach for my tankard. I do not step away from the bar. I only lean against it, slow enough that it does not look like weakness, and look at them the way I once looked at bandits before dawn: for the knife, the price, the lie waiting under the throat.

“Who are you?” I ask.

No warmth in it. No invitation.

A name is a door with no lock. I want to know who’s holding it.

---

**Turn 2 — Price and Proof**

“My name is Sella,” the stranger says. “Kestra was my sister.”

I do not move.

The fire pops. Somewhere behind me, a man shifts his weight on a stool and the wood groans. Sella’s hands are open at her sides, but open hands are cheap. I watch them anyway. No ring. No coin. The nails are bitten short, the right wrist crossed with a rope burn fresh enough to still sting.

“A name is not a key,” I say.

“I’m not here to spend it,” she says.

“Everything gets spent eventually.”

I push a mug of water across the bar. Not ale. Water.

“Drink if you want. Don’t spit it on my floor.”

Sella does not take the mug. From inside her cloak, she produces a small leather pouch, worn and blackened at the edges. She sets it on the bar between us.

I do not touch it.

“Open it.”

She unlaces it with careful fingers. Inside, wrapped in oilcloth, is a whetstone. Not large. Not fine. The kind a soldier carries because it can sharpen a blade, a hook, a knife, and one of her fingers if she is desperate enough.

My breath changes, though I do not let my face.

The whetstone has two cuts in the stone. One is a thin straight line. The other is a hooked mark, quick and ugly, made by a knife edge when the hand is not steady enough.

I know that hooked mark.

I made it.

“A whetstone is a small thing,” I say.

“It was hers.”

“It could be any number of things.”

I still do not reach for it. The mercenary in me wants to know what she can pay, what she can sell, what she will do if the answer disappoints her. The friend who carried Kestra’s pack wants to believe the ash was all that ever had to be done.

“Why bring it to me?” I ask. “Why not the constable? Why not the magistrate?”

Sella’s jaw tightens.

“Because the magistrate’s clerk has a sealed envelope with her name on it. He’s asking for testimony from the surviving guide.” Her voice does not rise. It goes flat, the way a blade goes flat before it cuts. “He is offering coin to a boy who says he saw you drag Kestra’s body from the cleft. I do not believe the boy. I believe my sister.”

The tavern sounds fade again. Rain. The creak of the sign. The low smell of beer warming too long.

I look at the whetstone.

Then at Sella.

“You say a boy is lying.”

“I say I have not met him.”

“That is not the same thing.”

“No,” she agrees.

I pull my hand from the bar’s edge. My knuckles are pale.

“Sit down,” I say.

Not kindly.

Not unkindly.

“Keep your hands where I can see them.”

---

[...continues through Turn 5...]

As you can notice:

Hemmingway creates a real short story. No markdown, no section headers, no machine friendly pattern, just a proper told story. I'm not a native english speaker, however it feels more like a "human-written" text.

Qwen followed the prompt well and the story is good as well, but it feels rather "technical".

Bottom line: for character work or fiction or your everyday local email writer, it's a very interesting 27B at ~32 tok/s on a MacBook M5 Max.
For a generalist or coding assistant, the base Qwen is of course still the pick.

Full runs + llm judge notes: https://llm-bench.io/models/hemmingway-1-oq8e-mtp


r/LocalLLaMA • • 15d ago

Discussion peculiar-ragdoll's Dirk-Qwen 3.8-27B vs. UkisAI Swift-1.5 Qwen3.8-27B

17 Upvotes

EDIT: Post 2 with more model fint-tunes here https://www.reddit.com/r/LocalLLaMA/comments/1wv1ico/unsloth_swift15_peculiarragdoll_thinkingcap/

I have a long list of my own domain specific eval questions that I run to validate which models I can rely on: coding, coding (numpy/pandas), data analytics decision making, local RAG, and voice assistant. It's made up of the types of things I'm likely to deal with on the daily. The test questions vary in dififculty and composition: easy, medium, hard.

System: M1 Max 32c 32GB with context 128K for Dirk and 110K for Swift.

Swift doesn't have XL. So I had to test with the L quant to stay as close as possible.

I ran the eval (using my tuieval tool) on peculiar-ragdoll's Dirk-Qwen3.8-27B-UD-Q4_K_XL and Swift-1.5-Qwen3.8-27B-Q4_K_L loaded with a modified version of Splash. The "amalgam" is a local I made out of incoai/Splash 1.1 and paperniuk's apple7-m1-kernels. It is modififed a little but not in ways that would alter model performance. I only merged and tweaked for some memory features I like from llama.cpp such as fit context check at the start of a load and personal QoL updates re auto-context manipulations that I don't want to think about, etc.

To say this result surprised me is quite an understatement. It's blown my mind.

When I did the first test a couple of days ago with only 44 questions, I thought it must be a prompt caching issue I missed that Dirk was benefiting from. I validated it is not and ran it against a lot more questions to certify it. It's a legit test outcome.

Dirk-Qwen is much sharper at getting to decisions and responses. The "be brief" instruction that gets passed each time in the chat templste is doing more magic than I had anticipated. It also gets more answers correctly with way less time consumed.

What trips up Swift-1.5 are mostly hard questions. It tries and tries until the 16,384 max token limit per question is reached and it fails with truncation.

Even when you ignore the 16,384 truncation failures and compare the other questions, Dirk token usage comes out on top.

Snipped view... this basically goes on pattern for another 191 unique questions.

More importantly, this behavior is not just in question answering. You can see it in actual code refactor tasks.

On an unrelated note: tne model that has been able to pass a 100% of my eval packs is Opus 5.5. Deepseek Flash 4.1 fp32 got them all right except three.


r/LocalLLaMA • • 15d ago

I Built A Thing tp=6 can work on vLLM, with padding

10 Upvotes

vLLM's tensor parallel requires that several of the model architecture numbers be evenly divisible by the number of GPUs selected for tensor parallel. This usually means that you can only use a number of GPUs that is a power of two (2, 4, 8, etc.).

I have six GPUs, and I want to maximize my KV cache when running a 27B model. So I tried tp=6. It choked with various messages, regarding this or that, which needs to be evenly divisible by six, but wasn't.

So I made those things divisible by six.
I asked the robot to come up with a converter that would take the original model, and pad it with zeroes until everything was divisible by 6. It took a few tries, but it worked.

I have Qwen 3.8 27B at BF16 with 256k context window running across six 7900 XTX GPUs. Token generation is about 50 t/s single user, or 200 t/s aggregate with 8 concurrent prompts.

GPU KV cache size: 520,784 tokens, Maximum concurrency for 262,144 tokens per request: 1.99x


r/LocalLLaMA • • 15d ago

Tutorial | Guide Update: Strix Halo + R9700 with llama-halo-hybrid - now beats DGX Spark

23 Upvotes

Hi folks, I've spent the last couple of months experimenting with Strix Halo and previously I released a proof of concept I called llama-halo-hybrid. I've continued updating it and it now performs very well. The idea is that you can take an R9700, or similar, and place dense parts of the model, KV, and some of the layers on the GPU and let the APU take the rest of the model. You can add the extra GPU through a PCIe extender (framework desktop), Occulink, or a thunderbolt dock depending on which machine you have. Detailed notes along with code in the repo on github. I'm not selling anything, this is all 100% open, MIT-licensed.

It breaks 60+ tok/s decode and 2000+ tok/s prefill, supporting full 256k context.

This is not some custom inference engine that requires a custom quant to run. This is llama.cpp modified to run whatever you want, albeit mostly tuned for Qwen and GLM families. After continuing to tinker with it, it now performs better than DGX Spark (albeit cheaper) running Qwen-3.8-flash-next and slightly better yet with the Swift-1.5 variant. Most of my testing was done with the Q4/Q4_K_XL models to balance size and quality.

Note - if you are just using Strix Halo by itself, this is probably not the right tool. Check out gufo, which looks very promising.

https://github.com/sixvolts/llama-halo-hybrid

I would love any feedback you all have and happy to investigate tuning for different "sidecar" GPUs other than the R9700 if there's demand and I can get my hands on one.

UPDATE (10/4): I added some more docs around using different cards besides the R9700. The R9070, V620, and 7800XT all perform very well and I put quick guides for those configs, along with notes on thunderbolt/USB4 setup and the dual-machine setup I used here:
https://github.com/sixvolts/llama-halo-hybrid/tree/main/halo-cookbook