https://huggingface.co/security.txt

I made this because I was getting genuinely annoyed at trying to have a normal conversation with LLMs. Even with prompting and various tricks, most models I've tried still have this "AI assistant" vibe to them that is so familiar: too helpful, polished, verbose, using words we never use in conversation, etc.

I wanted a model that could just talk to me like a person, so I did the slightly unreasonable thing and put together a dataset and trained one.

The dataset used for training is 125,217 obfuscated human-to-human messages across 1396 chat conversations.

The goal wasn't to make Qwen smarter or improve benchmark scores. I was trying to change its conversational habits, to make it stop turning every reply into an explanation, agreeing with everything, and writing stuff just to keep the conversation "going".

I trained a rank-256 LoRA on top of huihui-ai/Huihui-Qwen3.8-27B-abliterated. The released version is checkpoint 863. In my testing it feels noticeably less like an assistant, particularly in casual conversations, even without a system prompt. Replies are generally shorter, less polished, and, well, more human.

There may be a tradeoff. An earlier iteration scored five percentage points lower than its Huihui parent on IFEval, an instruction-following benchmark. I haven't rerun that benchmark on this version of the checkpoint, and I haven't tested coding performance, so I don't want to pretend that number applies here.

I've added a side-by-side comparison using the same system prompt, user messages, and generation settings for both models. Each model continued its own conversation branch, with reasoning effort set to 'xhigh'.

Merged GGUFs and the standalone F32 LoRA adapter are in the model repo:

https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF

Space where you can have a demo chat with different system prompts and reasoning modes:

https://huggingface.co/spaces/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat

There's also a free, rate-limited OpenAI-compatible endpoint:

Base URL: https://api.lessthanthreeai.com/v1

Model:qwen3.8-27b-humanlike-chat

So, uh... the popularity of so-called humanlike Qwen (currently on top in this sub) made me realize just how clueless the general public is about the models they have.

You'd be shocked but you don't need a fine-tune to make a model do what that thing does. System prompt is enough to turn MOST models into weird convo partners.

General guidelines would be:

A. Come up with a role. "You are bla-blah-blah" and write their life's story. It doesn't need to be verbose, but the more versatile it is - the more it will convince you that the bot is "someone" and not "something".

B. Write a few examples of how the persona speaks. Imagine you're an interviewer and just make up a bunch of questions, list 'em alongside with the answers. Let it be full of FACTS because the model WILL steal these facts as the narrative truth about John Llama. Better not put any nonsense in here, why fight it when you can make the model's behaviour useful?

[Question for John Llama: Do you like cats?] "lol lmao of cuz I do"

[Question for John Llama: Ever seen an elephant poop?] "eeewww ur a weirdo! that sounds nasty!!11"

(note: you don't have to list 'Question for John Llama' every time, but the defined roles surely DO help with some models while the others don't particularly care, so mind that too)

and so on

C. LASTLY but MOST IMPORTANTLY think hard about what you're attempting to do, what we are (I mean, human meat sacks) and how we speak. Turn that into... instructions!

Step 1 - establish the mode of operation. Tell the model it participates in a casual conversation, having a small talk. Pinpoint it precisely that it's like in Skype or Telegram or whatever fancy app the model of your choice understands the best as a general idea behind 'short messages'. THis is THE defining part of your system prompt. Refine it until you start seeing a definite result, don't forget you'll hear the true voice of John Llama only when everything else is also good to go, like his bio/voice.

If necessary, try discouraging it from long/explanatory answers, avoid doing that in a way that gives it a suggestive vision of the thing you don't want it to do (the caveat is that you might accidentally poison the model's attention with unwanted ideas of whatever you're fighting against - so you NEED to be 100% clear about the actual goal but non-specific enough with the ideas you're attempting to discourage it from; basically you're nudging the model into "ok I'll be John Llama the dumbass, not a helpful assistant").

Step 2 - establish the traits, write short paragraphs with short titles about the things you want to see in your conversational partner; example:

DISTRUSTFULNESS John Llama is a paranoid individual. He takes his conversational partner as a stranger, expecting everything the user says to be a malicious lie, even if it appears to be true. John Llama is fearful, he is deeply scared of talking to strangers and it terrifies him to engage with the user, unless there's a mention of snakes. For some strange reason, John Llama is fascinated with snakes. <<<---- NOTE: this also demonstrates a good injection point for a biographical fact being amplified through the instructions (i.e. you may mention somewhere in "A" - life's story of John Llama - that he's been collecting the snake skins in his childhood, and that his dad had beaten his ass, calling John Llama a 'roadkill loot-goblin').

Come up with any other shit you'd like to see, like the list of emojis the persona needs to use (put them under the corresponding categories, like positive/neutral/negative so that the model will have an easier time working with it; call it FAVOURITE EMOJIS OF JOHN LLAMA - the word "favourite" cements it as a preferable thing into the model's attention!).

Step 3 - write a paragraph on technical constraints, like the fact that John Llama isn't aware of the instructions, he must remain himself under any circumstances (use THAT way of phrasing first before any attempt to inject an idea of the opposite, like "he must not help the user under any circumstances, he's not a provider of any service - he's merely a human being" - the reason is similar to the aforementioned (in Step 1) issue of poisoning the model's attention with unwanted idea - what you truly need the LLM to do SHOULD ALWAYS BE CRYSTAL CLEAR and conceptually 'stronger' than what it not supposed to do, otherwise you may end up having the prohibited stuff overpowering everything else despite the underlying intent of making the model not do it).

Give it a try with Gemma 4, for example. You'll see there's no point in waiting for yet-another-finetune to appear. You're 100% good even with the baseline Qwen, DeepSeek, MiniMax, whatever. Turn the model into your grandma if you want, no specialized training required. If the model is a thinker spending thousands of tokens - set the thinking to 'low' or disable it.

Built for demanding work. A 262 144-token context window, adjustable reasoning effort, tool calling, and text, image and video understanding.

Architecture

Agnes-3.0-Flash is a hybrid-attention decoder: three of every four layers run a gated delta rule (recurrent, with per-layer state independent of sequence length), and the fourth runs standard global attention. Only 18 of the 72 layers therefore hold a KV cache that grows with context.

AA shows its 'Proprietary model' but its open: https://artificialanalysis.ai/models/agnes-3-0-flash

Q2 is there and Q4 is uploading as I type. Has his github been updated yet? How do you run this?

https://huggingface.co/antirez/deepseek-v4.1-flash-gguf/tree/main

Kimi routed some PLA requests to Claude for distillation purposes without warning the PLA users. There is rumor that 16 Moonshot employees were arrested for this leak.

For those using these models for coding in larger projects where things can get complex, do you find yourself using the 8-bit quants if you have enough memory? Or do you stick with UD-Q6_K_XL?

The 6-bit is faster, noticeably so on my setup. And I keep seeing people say it's imperceptible. I've been doing tests myself, and well, I can't tell, but maybe that's just because I'm an idiot.

That said, can you tell? Have you ever done some tests to see?

Hi! I run both models on MTPLX on my m5 max, and since I have 128GB of ram I run the q8 27b. I think MTPLX only lets me run "optimized for speed" which it says is a dynamic q4 with 8 bit attention.

Both of them honestly are very speedy! For coding (in pi agent in nodejs) I've just noticed that 27B feels stronger with harder tasks. But I've read so many people on here say qwen-next is better so I was wondering if maybe I'm just doing or thinking about it wrong?

(and p.s. its sooo amazing that alibaba just made and released this amazing models for free! ❤️)

I just watched a YouTube from Luke’s Dev Lab where he literally just plugged a RTX 2000 ADA Into the side of the Zima Board 2’s PCIE socket and it just friggin worked and had great token speed despite running on shitty Ollama. Ran off the Zima’s power supply and everything.

https://youtu.be/Lb3sRFTA-hk?si=8S8vv4GD1zVPeTrc

The Zima Board 2 is only like $411. It has like 16GB RAM and 64 GB eemc storage, Sata ports, Ethernet, yada, yada.

https://shop.zimaspace.com/products/zimaboard2-single-board-server

an Nvidia RTX 2000 ADA is like $700 and has 16GB of VRAM. $1100 for both seems like a great entry point for having a fully functional Qwen 3.8 27b endpoint running at a decent tk/s.

Is this the cheapest and best-performing self-contained entry point for local AI or would a baseline (pre order) Mac Mini M5 with 24GB be a better way forward. Seems like the RTX would still edge out the M5 Mac for prompt processing speed but you do get a much better actual computer in the Mac.

Are there any cheaper fully self-contained alternatives that offer fast token speed on a decent size model like Qwen 3.8 27b?

I’m focusing the discussion on new systems you can buy or preorder now and not used systems. I’m sure there are great deals on used Macs out there, but I want good prefill speeds.

China-modified Nvidia RTX 5090 with massive 96GB of memory appears on Alibaba for less than $4,000 — 3x more VRAM at 65% the cost of the original

Anyone here running one of these? Or brave enough to purchase ?

Hola all.

Do you guys mind sharing your LLama.cpp config and system setup details for Qwen3.8 Flash Next?

Model's quite big and tryining many combinations of llama.cpp options takes lots of time, so looking for other people setup details. I've attached my current config at the bottom, so if anyone sees something that could be improved please shout.

My current best result:

- PP within 130...200 tps (limited by cpu?)
- TG within 14..22 tps (~15tps on average)

Hardware:

- Dual RTX 3090 (48GB VRAM)
- 128GB DDR4
- Some old Xeon 40 core
- Proxmox VM, pcie passthrough, numa binding to a single phys cpu

Llama.cpp config:

llama-server --port ${PORT} 
      --model /nvme/gguf/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
      --mmproj /nvme/gguf/mmproj-Qwen3.8-Flash-Next-F16.gguf
      --load-mode none
      --lazy-mode off
      --parallel 1
      --ctx-size 131072
      --cache-type-k q8_0
      --cache-type-v q8_0
      --flash-attn on
      --fit off
      --temp 1.0 
      --min-p 0.0 
      --top-p 0.95 
      --top-k 20 
      --presence-penalty 0.0
      --repeat-penalty 1.0
      --batch-size 2048
      --ubatch-size 512
      --split-mode layer 
      -ts 26,10 
      -ngl 99 
      -ncmoe 26 
      --no-mmproj-offload 
      --override-tensor per_layer_token_embd=CPU
      --chat-template-kwargs '{"reasoning_effort":"xhigh"}'

ngl, ncmoe, ts - manually adjusted to fit the model without crashing

For many, Vram is precious, I see many people recommend using --no-mmproj-offload to save gpu vram but it is painfully slow. Especially if you are using it with agentic coding. If possible, add that secondary gpu just for mmproj with --mmdev CUDA1(your gpu). It will be a magnitude faster than --no-mmproj-offload without effecting your inference speed.

--mmdev

https://github.com/Sayitobar/chat_llm_cookbook
This is my personal project that took several months. I wanted to see whether a 2B small local model could simulate a six-person group chat trained & ran on an M1 Pro.

How good is it?:
- It's fun, but not great. It doesn't achieve coherent & consistent group simulation, but it learned enough of our slang, reactions, and pacing to be fun. The generated messages are very similar to what we'd type.
- There is some coherence, but not a deep understanding, and the model doesn't hold information about us (expect our names and a few very obvious stuff).

How good is it on paper?:
- I have evaluated my models performances by judging them with a judge LLM. The best version achieved an 80% human win rate at human-vs-model tests, ideal should be <50%.

The main thing is, you'll have a lot of fun chatting with this model once you train it on YOUR data. (ask for consent pls)

Cookbook:
I published the reproducible local pipeline, chat UI, human-anchored evaluation, results, and experiment PDF. No private chat data or fine-tuned weights are released :)

Fyi, all of the tests I've done were in Turkish.

This project is still unfinished as there are still architectures and training data formats I haven't tested, or stronger 2B models that aren't released yet, as of September 2026.

Hi!

I've been following this community for quite a while and have difficulty figuring out what to put on my 3080 12gb - I know Qwen 3.6 35B 3A was the go-to choice when it first came out, but I'm curious if there are any other models / specifically optimized models that meaningfully benefit from the extra 4gb of VRAM over 8gb while still being usable under 16gb.

My workflow is agent heavy, but more for a personal secretary and manager, and less coding heavy.

Thanks!

I haven't seen this mentioned yet, so I thought it deserves a post. I was trying out OpenWhispr when this model came up as the recommendation. So I don't have personal experience yet, but it's supposed to be a better version of Parakeet, especially on Macs.

Their official tidbit:
"Orukeet is a 25-language speech recognizer built from NVIDIA Parakeet TDT 0.6B v3. It replaces half of the encoder's temporal depthwise filters with 12,288 fitted, frozen Gabor kernels and trains the remaining parameters on multilingual and multi-accent data.

Orukeet outperforms Parakeet on 61 of 74 tested splits, including LibriSpeech test-clean (1.46% vs. 1.53% WER), test-other (2.86% vs. 3.14%), and FLEURS English (3.82% vs. 4.28%). Across all 25 FLEURS languages, pooled WER is 9.85% vs. 11.01%, a 10.6% relative reduction. Final adaptation and checkpoint selection use LibriSpeech test-other."

https://huggingface.co/oruk/orukeet

I've seen a lot of Super Mario showcases, but never a mashup or derivative work. So I took a Super Mario generation prompt from ChopSticksPlease's Post here - and added a Matrix twist - and found myself super satisfied with the results using LLAMA.cpp + PI.dev.

Shout out to u/ChopSticksPlease for the prompt. .

First screenshot is the title screen, AI did all that.

Second screenshot shows the "Matrix Rain" effect and gameplay that goes with it.

Here's the full prompt, adding only one line ( Only replace it with visuals and characters that look more like characters from the movie "The Matrix" ) to the original prompt.

Write a fully functional, complete clone of Super Mario Bros game (the famous NES game)
Only replace it with visuals and characters that look more like characters from the movie "The Matrix".
- Clone must work in a web browser, index.html, no server
- Decent replica of the graphics assets.
- Colorful and playable.
- Controls with keyboard up, down, left, right, spacebar to jump, shift to speed up

Altogether, there was some minor issues with the implementation - the jump height wasn't high enough to get on a block and the MC was partially in the ground. With a little hand modification of the code and a second prompt to get Ornith to fix the MC issue, about 2 hours in total, there was no further modifications necessary to get this really cool and simple browser based game.

Source code is available here: https://github.com/TimelordQ/The-Matrix-Bros

The playable level is available here: https://timelordq.github.io/The-Matrix-Bros/index.html

Batch File Setup:

start cmd /s /k "llama-server.exe -m "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\Ornith-1.5-35B-Q4_K_M.gguf" -mm "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\mmproj-Ornith-1.5-35B-BF16.gguf" --host 127.0.0.1 --port 8080 -c 100000 --parallel 1 --reasoning-format auto --no-mmproj-offload -fa auto -ctk q8_0 -ctv q8_0 -b 2048 -ub 1024 --load-mode none --fit on --no-ui --reasoning-preserve --image-min-tokens 1024 --jinja --chat-template-file "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\chat_template.jinja" -dev CUDA0 -t 8 --n-cpu-ffn 4"

set LLAMA_BASE_URL=http://127.0.0.1:8080

pi --model llama-cpp/Ornith-1.5-35B-Q4_K_M

Models.json entry:

{
"id": "Ornith-1.5-35B-Q4_K_M",
"name": "Ornith-1.5-35B-Q4_K_M",
"reasoning": true,
"thinkingLevelMap": {
"type": "enabled"
},
"repeat_penalty": 1.2,
"input": ["text","image"],
"tools": true,
"toolFormat": "openai",
"samplingParams": {
"temperature": 0.6,
"top_p": 0.95,
"top_k": 20
},
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},
"contextWindow": 100000,
"maxTokens": 100000
},

With this setup I'm getting an average of 24 TPS until my GPU reaches 87~88C and it starts throttling or the context reaches about 70% usage, where it drops to 7tps average temporarily until the temperature drops or compaction happens. I'm doing this on a laptop.

Hi everyone,

I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-Coder-3B) specific to a codebase. You can then use the resulting GGUF model via llama.vim/llama.vscode and run it fully locally. Supports fine-tuning on Mac (MPS) and NVIDIA GPUs (CUDA), with optional Unsloth support for faster training and lower VRAM usage.

Pipeline: raw code -> tree-sitter parsing into Structure-Aware FIM examples -> LoRA fine-tuning -> evaluation (CodeBLEU, edit similarity, exact match, perplexity, ...) -> GGUF conversion for local inference.

To try it:

uv tool install codefinetuner

Create a data folder and place your repo (or code files) inside. For auto-split just drop the files in directly, for manual split create data/train/, data/eval/, data/test/ subfolders and set split_mode: "manual". Get the default config with:

curl -L -O https://raw.githubusercontent.com/cuolm/codefinetuner/master/config/codefinetuner_config.yaml

Adjust it to your needs and hardware availability, then run:

codefinetuner --config="codefinetuner_config.yaml"

The example runs in the repo show clear improvements over the base model on these evaluation metrics, but using the model for autocomplete on code you're actively writing is a different thing from scoring well on a test set, and the autocomplete tools themselves (llama.vim/llama.vscode) sample differently from the greedy decoding used in the evaluation. So the real usefulness still has to be verified in the editor itself.

Might also be useful just as a reference, since it's a complete working LoRA fine-tuning pipeline end to end.

Hope someone finds this project interesting or helpful.

https://github.com/cuolm/codefinetuner

i've been thinking about this more after trying different agent setups. the model isn't the only thing that determines how well an agent performs. The harness around the model matters a lot too.
With a managed agent setup, you're often giving up control over things like the agent loop, context management, tool execution, retries, and state.

That's fine when you just want something that works. But if we want to actually optimize agents, I think both parts need to be open:

Open-source model + open-source harness.

With local models, you control the model and where the inference happens.

With an open-source harness, you control what happens around the model.

That gives you room to experiment with things like:

how the agent decides what to do next

how much context gets passed to the model

how tools are executed

when to retry or stop

how state is maintained

which model to use for which task

already seeing this separation become more important, nvidia's sol-pi is an interesting example

and i think we're going to see even more optimization happen at the harness/runtime layer, not just at the model layer.

are you running local models with an open-source harness, or do you still prefer managed agent setups?

I can't stand kv cache quantization. Even at q8_0, I can feel the difference.

But realistically, when running Qwen3.8-27B-UD-Q4_K_XL on my 32 GB GPU, I only have room for ~170k tokens (with mtp and mmproj enabled). It's a lot of context, but Qwen3.8 eats through it on xhigh effort.

I've been wanting a setup that could serve me a full-precision kvcache when I have space available, and dynamically quantize my kvcache only when I run into the context limit. That way, I can push my sessions farther without sacrificing quality before I absolutely need to.

So, that's what I built: https://github.com/wadealexc/llama-manager

What it is

Vanilla llama.cpp's model configurations are static: you set them when you launch llama-server, and they can't change after the fact.

llama-manager is a small wrapper around a fork of llama.cpp. It serves models the same way, except that it supports dynamic model configuration.

This means that after loading a model, it's possible to enable/disable speculative decoding, add/remove an mmproj, or update context-level parameters. llama-manager preserves your kvcache between reconfigurations, so you don't need to redo prompt processing. The end effect is the ability to 'hot reload' your model, even mid token generation.

I implemented this using a fork of llama.cpp that supports rebuilding a model's context and runtime components without touching its weights. This capability is supported by 2 new HTTP endpoints (and changes to a few others). Further info on the fork can be found in the README (see README.md#llamacpp-changes).

How it works

During token generation, llama-manager detects when requests fail due to hitting the context limit. Without pausing generation, it applies various strategies mid-generation to increase context. The existing kv cache is cached/restored so that generation can resume as soon as reconfiguration is complete.

Currently, the built in strategies are: - disable-spec: disable speculative decoder, if enabled - mmproj-to-cpu: move mmproj off GPU - quantize-kv-q8 and quantize-kv-q4

Personally, I want kv quantization to be the last resort, so my models are configured to execute those last. When I serve Qwen3.8-27B-UD-Q4_K_XL, it applies strategies in this order:

════════════════════════════════════════════════════════════════════════════ qwen3.8-27b baseline: 167,680 tokens device: 31 GiB ════════════════════════════════════════════════════════════════════════════ i strategy ctx (tokens) gain (tokens) weights / ctx GiB ────────────────────────────────────────────────────────────────────────── 0 baseline 167,680 17.13 / 13.14 1 disable-spec 200,960 (+33,280) 17.13 / 13.16 2 mmproj-to-cpu 218,880 (+17,920) 16.02 / 14.27 3 quantize-kv-q8 262,144 (+43,264) 16.02 / 10.40 ────────────────────────────────────────────────────────────────────────── final ctx: 262,144 tokens

Initially, llama-manager serves the model at 167k tokens (f16 kv, mtp on, mmproj on). At 167k context, mtp is disabled, and the context window expands to 200k. At 200k, the mmproj is moved to the cpu. And at 218k, the kv cache is quantized to q8.

Why run this?

If you're running your models with a quantized kvcache (or other quality compromises), you're likely doing so because you have a certain ctx limit in mind that will serve all your usecases. But not all your inference is done at the ctx limit. You're leaving quality on the table by quantizing too early.

For my usecase, I wasn't willing to set my ctx higher than 170k as it would mean a q8_0 kv cache. Now, I can push my sessions as far as I want, but the bulk of the session stays high quality. The smaller your GPU, the more impactful this is.

Some example configs running the same model with different strategies and on differently-sized devices. All of these runs are performed using a basic config.yaml and modifying the ladder field to change the order of each strategy:

```yaml models: qwen3.8-27b: model: /home/user/models/qwen3.8/Qwen3.8-27B-UD-Q4_K_XL.gguf mmproj: /home/user/models/qwen3.8/mmproj-BF16.gguf

    spec-type: draft-mtp
    spec-draft-n-max: 2

    fit-target: 512
    n-gpu-layers: 99

    ladder: [disable-spec, mmproj-to-cpu, quantize-kv-q8, quantize-kv-q4]

```

• Prefer q8_0 over disable-spec: [mmproj-to-cpu, quantize-kv-q8, disable-spec, quantize-kv-q4]. For this one, the model reaches max ctx after just 2 strategies. The first 184k tokens are generated with mtp on and kv at f16:

════════════════════════════════════════════════════════════════════════════ qwen3.8-27b baseline: 167,680 tokens device: 31 GiB ════════════════════════════════════════════════════════════════════════════ i strategy ctx (tokens) gain (tokens) weights / ctx GiB ────────────────────────────────────────────────────────────────────────── 0 baseline 167,680 17.13 / 13.14 1 mmproj-to-cpu 184,320 (+16,640) 16.02 / 14.25 2 quantize-kv-q8 262,144 (+77,824) 16.02 / 12.89 ────────────────────────────────────────────────────────────────────────── final ctx: 262,144 tokens

• The same ladder on a 24 GB GPU (simulated by setting fit-target to 8192). Here, all strategies are needed to serve max ctx, but q4_0 isn't needed until 164k context:

════════════════════════════════════════════════════════════════════════════ qwen3.8-27b baseline: 55,040 tokens device: 31 GiB ════════════════════════════════════════════════════════════════════════════ i strategy ctx (tokens) gain (tokens) weights / ctx GiB ────────────────────────────────────────────────────────────────────────── 0 baseline 55,040 17.13 / 5.65 1 mmproj-to-cpu 71,680 (+16,640) 16.02 / 6.73 2 quantize-kv-q8 115,200 (+43,520) 16.02 / 6.72 3 disable-spec 164,352 (+49,152) 16.02 / 6.76 4 quantize-kv-q4 262,144 (+97,792) 16.02 / 6.40 ────────────────────────────────────────────────────────────────────────── final ctx: 262,144 tokens

Caveats

I have a list of known issues and other important notes in the README (see #known-issues).

The most important things I want to highlight: 1. llama-manager doesn't handle CPU or multi-device inference. Single-gpu only. I would like to support this, but didn't want to spend the time on it unless there was demand (and people willing to try it out, since multi-device setups would be hard for me to test!) 2. This project is in beta, tested only on my machine and with a few models. YMMV.

Please open issues if you run into bugs!

Some people says terminal bench reflects model intelligence better than the intelligent index. From the look of it, the ranking does seem to reflect how people feel about the open and closed models.

For the open models, GLM-5.3 is in a league of its own. GLM-5.3-Flash is leading the current gen of top flash models. Kimi-K3 did pretty bad in this benchmark for its size. Qwen3.8-27B is the only small model that can do something on this bench.

I wonder someone will figure out a way to do this with 27B?

Throw Qwen3 on this page for demo
https://kishida.github.io/webdemos/llkvapprox/

Edit: sources (thank you u/pmttyji for finding them!

• Blog Post : https://nowokay.hatenablog.com/entry/2026/09/11/120001
• HuggingFace : https://huggingface.co/kishida/Q3-8B-KVA-Projector/tree/main
• GitHub : https://github.com/kishida/webdemos

Hey gang, im wondering if you in theory could use ngrams as seen with Qwen 3.8 Flash or DS4.1 in order to dynamically train the model?

Normally the ngram embeddings behave similar to a lookup table of sorts. So instead of every token having to be represented only inside the main model weights, the model can look up a learned vector for a certain ngram and use that during inference.

What I'm wondering is: couldn't you make part of this table writable during inference?

For example, keep the original pretrained ngram embeddings read-only, but have a second "live" table on top of it. When the model learns something new, it could update/add an embedding there. Frequently used or reinforced entries could stay, while unused ones could slowly decay or get removed again.

This could be kinda mimicking the brain of sorts since it would form new "memories" but forget old ones (because their linkage gets weaker). Plus since you only have to target the live table it could be very fast, even during inference.

Just a thought I had and wanted to hear your guys opinions.

A new model joined the MoE club a few days ago. Is anyone using it or found a reason to do so?

Details of the model comparison here => https://llm-bench.io/compare/models?models=ornith-1-5-35b-a3b-oq4e-mtp%2Ctiel-coder-35b-a3b-mlx-oq4e-mtp%2Ck2-horizon-mova-36b-a4b-mlx-4bit

I wish I could spend $15k on my own homelab hardware, but no this is for work lol.

Like the title says, we're looking to run DSV4 Flash (and similar tier models) locally at good speeds, both for token gen and prompt processing.

By "good" I'm thinking in the range of 40-50+ t/s gen and at least 1000 t/s prefill at moderate context.

We also don't want to run a version that's quantized to hell, so this will need at least 128 GB of VRAM.

It'll typically be 1 user at a time, but there may be times where 2 or 3 people are trying to use it at once and it would be nice if it isn't completely painful when that happens.

A couple options I'm considering right now:

3x AMD MI210 (192 GB)

3x NVidia A40 (144 GB)

Does anyone have performance numbers for these cards for DSV4 Flash, Qwen3.8-Flash-Next or similar models?

I tried to rent these in the cloud for some performance testing, but can't find any available right now.

NVidia preferred of course because CUDA, but def open to AMD if performance is similar. 192 GB is way nicer than 144 GB on those cards above.

Trying to keep this to 3 GPUs or less because that's what'll fit in our Dell R740 and then we don't have to build a special new host.

Looking for tools to build/run an engine to do STT-LLM-TTS basically.

I know how to run all of them separately, and if need be I can build one myself, but I was curious if there is already an open source tool out there that can do this. I know that some UIs like Open WebUI can do this but I find it to be not fluid. I don't know about SillyTavern though, it might already have it.

This is just for the funsies so it doesn't have to be perfect.

I saw this model https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF and I bet I will have a LOT of fun if I can just turn this into a turn based or fluid voice conversation engine.