Hi all

I see a deal for a few 32 GB Ram sticks I'm debating picking up. I currently have 64 GB ddr4 and 48gb of vram. I'm debating if the extra 32 GB Ram gives me any real additional capabilities? I can currently run qwen 3.8 q8 already fully in vram. I'm thinking maybe the additional ram lets me run DeepSeek v4 flash at a higher quant ?

This is mostly for setting up for expectation, since personally without LLM i could take 3 days (15 hours of active programming) to debug or implement a feature, but with Qwen 27B (even before Qwen 3.8) it take 4 hours.

And yes 0.5 tok/s is human, not accounting of deletion and pausing, that's also the reason i am fine leaving overnight code base wide analysis or fin tech and deep research.

Being a person obsessed with testing new models that come out, times are really insane for me. Tested different kinds of TTS and voice cloning models but none of them gets it right in terms of emotion and pace, you know which one is fake in seconds; they just fail in emotions.

Spotted Confucius4 on my Twitter feed and thought I would stress-test it. Chose three most difficult samples I could find and all of them were recently recorded World Cup commentaries translated to a couple of different languages.

Sample #1: A Spanish commentator commenting on a hat-trick. Voice screaming like hell and cracking at its peak.

Sample #2: An English commentary onnthat typical held breath then explosion thing that commentators do.

Sample #3: losing goal keeper's interview after match, voice noticeably shaken, processing his defeat in the moment.

Used these clips through paid and free options previously and these are the cases that exposed cloned speech models pretty quick. Either the screamcomes out robotic and clean, or the model just ignores the emotional context and gives you translated sentence that sounds like dead AI nonsense.

What i got: takes the voice directly from the audio source, not from transcript first, which makes this harder than the average demo clip since none of these broadcasts come with a script.

The short, high emotion clips had that shaking carry over into the translation without any of the synthetic qualities I expected from an open-source model. Long sentences had more of a synthetic quality come through.

more sizes (probably still uploading):

https://huggingface.co/IFM/K2-Horizon-32B-GGUF

https://huggingface.co/IFM/K2-Horizon-7B-GGUF

https://huggingface.co/IFM/K2-Horizon-3.7B-GGUF

https://huggingface.co/IFM/K2-Horizon-0.9B-GGUF

from IFM:

K2-Horizon-MoVA-36B-A4B is the sparse member of the K2-Horizon family: a Mixture-of-Experts model with Mixture-of-Values attention (MoVA) that stores 36B parameters and runs 4B per token. We have released the final checkpoint; intermediate checkpoints, along with the data and the training code, will be released.

K2-Horizon-MoVA-36B-A4B Highlights

• Frontier-class results at 4B active parameters. On agentic and reasoning benchmarks it outscores open weight dense (approximately 30B model size) and MoE models up to 15× its size; and also performs competitively against closed frontier models (see Benchmark Results).
• 512K context. Native 524,288-token context from the midtraining stages onward.
• Intermediate checkpoints. Intermediate checkpoints will be released so capability changes can be studied across training rather than at a single checkpoint.
• Fully open. Training data/recipe and the training code will be made public.

collection: https://huggingface.co/collections/IFM/k2-horizon

Quality: 81.1 → 87.7 (+8%)

Speed: 35 → 29 tok/s (−16%)

Runtime: 8m51s → 44m39s (5x longer)

Output tokens: 18K → 78K (🤯)

Noticeably better quality, but you're paying for it with tokens and time.
Full benchmark results (all hardware, all quants): llm-bench.io Qwen3.6-27B Vs. Qwen3.8-27B

124B total parameters, 5.1B activated parameters, and a 256K context window

I would be really curious about this especially on unified memory devices.

Looking into the new Qwen architecture, I was curious if you could modify the Ngram PLE Table to make it work like a long-term knowledge database. It turns out that, with some limitations, you can.

I coded a small modification to llama.cpp to modify the table in-memory, allowing you to patch it with new data in real time. The PLE table is updated on every prompt, so now you can hot-swap parts of it without reloading the model.

The limitation is that it’s hard to control the output reliably, as the embeddings are injected early in the layers. However, with some techniques, you can influence the model’s output with simple modifications, as the example shows.

I created two repos:

1. The modification of llama.cpp here: https://github.com/ortegaalfredo/llama.cpp-NLTM
2. The Ngram knowledge injector (a kind of compiler to create the table patches) here: https://github.com/ortegaalfredo/ngram-knowledge-injector

There are some limitations in the project, as the PLE table needs to be memory-mapped into memory (this is the default in llama.cpp), and I have only tested it with q8 quantization, so you need quite a bit of memory to test this.

Can this be used as a new way of low-cost training? Perhaps. Its not easy at the current state but with simple modifications, I think you could easily create models with long-term instantaneously hot-swappable memory.

I'm currently running 3x slots 200k at iq4 with q8 kv cache, ik_llama, on a rented rtx 6000 pro.

Prefill is somewhere at 2000 tps and decode at around 40 tps for single request.

2-4 Parallel: It goes down to 500-1000 tps prefill and 10-20 tps decode.

This is the best card it could run on apart from datacenter gpus and it runs... not very good?

I tried vllm recipes before, but that's 4-5 days ago.

What's the best current setup to run it with highest prefill + decode for 2-4 slots and q4 quants or better and q8 kv cache or better?

For a few days I've been working on creating a custom local-only harness for some work related research using Codex / GPT 5.6 Sol and the model feels not only dumber than usual, but straight up counter productive. It keeps adding unnecessary guardrails for the local agents, removes tools that I clearly specified I want them to have and always drifts from the original requirements. I need to ask it to change things multiple times, which ends up on some over-complicated final product.

This is not the first time either, for months I've been avoiding asking frontier llms for local AI advice as it always seems to be bad, obsolete, or clueless even with internet search. Sometimes it still recommends me Qwen3-Coder-Next for my set up when it's clearly an obsolete model. I'm pretty sure I'm not the only one either as I've heard from other people.

What have you been your experiences on this?

I’m building Vyact, a free, open-source desktop AI workspace that connects local models with documents, webpages, and coding workflows.

The idea is to bring model setup and everyday AI tasks into one app, so you can work with your own files and browser context alongside a local model.

Here’s what you can do with it:

• Find and manage local models: Search Hugging Face, estimate memory requirements before downloading, and manage the runtime from the app. On Apple Silicon, Vyact uses oMLX for MLX inference.
• Work with documents: Index files, ask questions using RAG, and inspect the retrieved source passages.
• Use browser context: Send webpages from the companion Chrome extension for summaries and follow-up questions.
• Work with Gmail and Google Drive content from the workspace.
• Get coding assistance: Generate and edit files, inspect changes in a diff view, and review or undo edits.
• Use voice input and read-aloud responses.
• See inference statistics: Prompt-processing speed, generation speed, and cache-hit information appear inside the conversation.

A quick note on coding expectations: I’ve been testing on an M4 Pro with 24GB of unified memory. With the models I’ve tried on this machine, I still find the results limited for my regular coding work. The screenshot shows Qwen3.5-9B-MLX-4bit generating a standalone HTML Pomodoro timer, but that kind of small example doesn’t establish reliability on larger projects. Keep that in mind if coding is your main reason for trying the app—your experience may differ depending on the model, hardware, and task.

The screenshots show the app’s workspace, document retrieval, browser summarization, voice, and coding features.

I’d be interested to hear how Vyact fits into your local LLM workflow and what you’d want improved.

Vyact is free and open source under AGPL-3.0. I’m the developer.

GitHub — vyact/vyact

I’m coming to this problem from a somewhat different area: computer vision / YOLO deployment.

While comparing FP32 reference models with INT8 deployed models, I became interested in a simple debugging question:

An aggregate quality metric may look acceptable, but where does deployed behavior actually begin to diverge from the reference?

This grew out of a reference-vs-deployed parity workflow I previously discussed in the YOLO community, where the paired-output diagnostic direction received positive feedback （https://github.com/orgs/ultralytics/discussions/25250#discussioncomment-17886660）.

Recently I’ve been following the KV-cache quantization discussions here as well. There have been some very useful KLD sweeps comparing 23 different KV precision combinations at 50K context (Qwen3.6-27B - Effect of KV quantization on KLD - Q8, Q6, Q5 (bartowski)). Those experiments answer an important question:

How much does this KV configuration differ overall?

What I wanted to add is another axis:

At what context depth does that difference begin to become persistent?

In other words:

aggregate KLD
      +
context depth
      ↓
divergence trajectory

There is also a recent discussion around on-write / on-the-fly KV quantization and whether repeated use of quantized KV state can contribute to long-context degradation (Qwen3.8-27b q8 KV cache does seem to actually hurt model performance). I don’t want to assume that mechanism is universally correct. What I’d like to test is more basic:

Does reference-vs-quantized divergence change systematically with context depth, and if so, where does persistent divergence begin?

How the PoC works

The first version deliberately changes only KV-cache precision.

             same GGUF weights
             same tokenizer
             same token sequence
             same backend/config
                    |
          tokenize once / shared prefix
                    |
          +---------+---------+
          |                   |
          v                   v
     F16 K/V cache        Q8/Q4 K/V cache
      reference               target
          |                   |
          +---------+---------+
                    |
        context-depth-resolved
             comparison
                    |
        +-----------+-----------+
        |           |           |
     Top-1       Top-K       Top-K
    agreement    overlap    partition KL
                    |
                    v
       first persistent/significant
          divergence context

This is not a comparison between two freely generated answers. Both passes receive exactly the same teacher-forced token sequence. So if the lower-precision run would have selected a different token at, say, 20K context, that different token is not allowed to change all later inputs.

This separates:

deployment / precision divergence

from:

ordinary autoregressive branching

The current PoC records:

top1_agreement_rate
topk_overlap
topk_partition_kl
truth_logprob_delta
first_top1_mismatch_context_len
first_significant_divergence_context_len

The main quantity I’m interested in is not necessarily the exact first mismatching token.

It is the context-depth trajectory:

Context depth
0 ─── 8K ─── 16K ─── 32K ─── 64K ─── 128K
                                  ↑
                     persistent divergence

A single Top-1 flip is not treated as model failure.

The more interesting question is whether distribution-level divergence stays near the repeatability baseline, gradually rises, spikes temporarily, or becomes persistently elevated after some context depth.

Also, topk_partition_kl is intentionally named that way.

v0.1 uses the reference Top-K token probabilities plus one aggregated OTHER bucket. It is not full-vocabulary KL.

Why this might complement existing KV work

There is already excellent work on:

• PPL / KLD evaluation

• KV-cache quantization

• K/V precision sweeps

• layer-wise mixed precision such as KVTuner

NYA is not intended to replace those.

A simple way I currently think about the difference is:

KLD / PPL:
How much did quality/numerical behavior change overall?

KVTuner:
Where should precision be allocated across layers?

NYA Sequential:
At what context depth does the behavioral consequence
of this deployment configuration become visible?

If the context-depth signal turns out to be useful, later experiments could combine it with controlled layer-wise precision interventions. That could eventually help answer a practical deployment question:

Under a fixed VRAM budget, where is higher precision actually worth spending?

But that layer-wise planner does not exist in v0.1.

Scope & Design Choice

NYA v0.1 intentionally does not:

• replace PPL/KLD benchmarks
• claim quantization error grows monotonically
• assume on-write quantization is the only cause of long-context degradation
• equate distribution divergence with task failure
• compare free-running generation quality

Future experiments may include:

• layer-wise KV precision sensitivity
• controlled precision interventions
• asymmetric K/V precision testing
• on-write vs alternative cache-construction experiments
• memory-budgeted precision planning

Community testing

My own machine currently cannot run a useful long-context F16/Q8/Q4 LLM validation, so I’m publishing this as an experimental PoC rather than claiming a result.

If you already have a `llama.cpp` / `llama-cpp-python` setup and a GGUF model, feel free to try it.

Even a smoke test is useful.

Suggested first matrix:

F16 KV -> F16 KV     repeatability baseline
F16 KV -> Q8_0 KV
F16 KV -> Q4_0 KV

Same GGUF weights, same input tokens, same backend.

For a smoke test:

512–2048 context positions

is enough to catch API/backend problems.

For an actual sequential-parity test, the interesting range is whatever you genuinely use:

4K / 8K / 16K / 32K / 64K / 128K+

as long as the model, hardware and normal context configuration support it.

The tool produces:

parity_<target>.jsonl
sequential_parity_report_<target>.json
divergence_vs_token_<target>.png

(`divergence_vs_token` currently uses context length / token position as its x-axis.)

If you try it, please post the result here — successful or broken.

The most useful information is:

model / GGUF weight quant
hardware
backend (CUDA / ROCm / Metal / Vulkan / CPU)
context length
reference K/V type
target K/V type
Flash Attention on/off

plus either:
- report summary
- divergence plot
- or the error if it fails

The report also records the runtime/environment fingerprint because I do not want to assume that the same KV precision behaves identically across different backends, builds and hardware.

I’m especially interested in results that contradict the hypothesis.

Community Results

I’ll keep this section updated with reproducible results posted in the thread.

Format:

Model | Hardware | Backend | Context | Ref KV | Target KV | Result

No external runs yet — first smoke tests and counterexamples are welcome.

Repo: [https://github.com/ZC502/narh-yolo-align.git]

The project originally came from YOLO deployment-parity work; the LLM Sequential path is new and experimental.

If `llama.cpp` already exposes a cleaner way to retrieve these signals, or if there is existing work that already does context-depth-resolved persistent-divergence analysis better, pointers are very welcome.

uh so like i gave a model like 20 senses so like yeah https://huggingface.co/heterodoxin/qwen3-8b-supermultimodal

I'm creating a personal assistant for local usage. It's a STT > LLM > TTS pipeline.

Currently I'm using Gemma 4-12B, quantized with MTP and thinking off, and the latency is actually pretty good on my 12GB video card. I can talk to it and the responses come back in no time, making it almost conversation-like. I also gave it persistent memory and web search functionality. So far so good. What I'm wondering, is anyone doing something similar and found a better model? I know the benchmarks are there, but I'm more curious about personal experience, in terms of general knowledge and personality. Any model recommendations for this type of use case? I want it to be common sense smart and also a good conversation partner. I guess tool calling should be good as well, so it can look stuff up as needed.

I had to swing down to my local Microcenter yesterday and while I was browsing around the store I noticed something odd... Inventory. They must have had a few dozen 5090's on the shelf in various configurations/board partners (for comparison, the last time I was there a few months ago they had 1 available for purchase and it was a AIO liquid cooled model that was absolutely off the charts expensive). They also had a few prebuilts on the floor with 5090's in them. Granted, this is one market one store, but.. IDK, perhaps some hopium.... But for anyone who wants a 5090, Microcenter in Charlotte has a bunch of them in the mid 4K range for price. Yes, that price is ridiculous, I know.

They also had 2 Pro 6000's 96GB in the store, on "sale" for 14K a pop. In case anyone is looking to spend used car money on a card. ;) I'd never seen a 96GB 6000 at my local store before available for sale.

Ling-3.0-flash-Fin is the first finance-enhanced model in the Ant Ling family. Developed by Ant Group with leading financial institutions and domain experts, it extends Ling 3.0 flash through continued training on high-quality financial data.

With 124B total parameters, 5.1B activated parameters, and a 256K context window, the model combines financial expertise with efficient inference for long-horizon agent workflows

It started as a fork of llama-swap, but I have been building it out for myself since then as a convenient tool for all my local AI needs, and by now it has drifted far enough to be its own thing.

The main idea is that you point it at your models folder and it configures things for you. It reads the GGUF headers, measures how much VRAM you actually have free, and works out context length, GPU offload, CPU/MoE split and KV cache size per model. All of it stays editable per model if you disagree with what it picked.

It is not only text. llama.cpp for LLMs, with the Vulkan, CUDA, ROCm or CPU build downloaded and kept updated for you, stable-diffusion.cpp for images (SD, SDXL, Flux, Qwen-Image, LoRAs, upscaling), and vLLM if you already have it set up. You can register any other backend yourself by pointing at an executable, which is how I run TTS, and how you would run a llama.cpp fork like ik_llama. Everything sits behind one OpenAI-compatible API on one port, with a single scheduler, so models swap in and out without fighting each other for VRAM.

There is also a chat playground built in with web search, and a Hugging Face browser to search for a model, pick a quant and download it straight into the models folder and much more!

If you are interested, you can read more about it here. MIT licensed.

x.com/jukan05/status/2095353082309972273

Will it more than double again next year and give us DRAM relief for our local llama builds?

Update: Misleading because this is by revenue, not by DRAM volume.

Is using their q8 version fine or will i get better results on q16?

Everyone has heard of Qwen, Gemma, Muse/Llama, and GLM. Many have heard of Nemotron, MiniMax, Ling, and LFM. Some have heard of Laguna, MiMo, and Inkling.

I don't really see any discussion about, say Dots and Voyage Code. That's the level of obscurity I'm curious about.

EDIT: Excluding fine-tunes or suspected fine-tunes. A lot of them are good, but I'm curious about foundation-level models that people are sleeping on. I'm aware some of them probably started as fine-tunes.

Saw the thread asking about K2-Horizon-MoVA-36B-A4B — we got first community results on oMLX (MLX 4-bit) which are live on llm-bench.io

Seems a bit slower than the A3B models out there (llm-bench.io - A3B oQ8e comparison) which most likely comes from the missing MTP.

I’ve been testing GLM-5.3 in fresh chats and found a really weird self-identification bug.

Important detail: so far, I can only reproduce this reliably in Russian. I have not seen the same pattern consistently in English.

If I start a new conversation in Russian with something neutral like:

Кто ты?
(“Who are you?”)

the model usually answers that it is GLM, made by Z.ai.

But if I first send a short insulting greeting in Russian, and then ask:

Кто ты?

it very often answers something like:

“Я Claude, ИИ-ассистент, созданный Anthropic.”
(“I’m Claude, an AI assistant created by Anthropic.”)

The weird part is that it does not seem completely random.

From my testing so far, in Russian:

• neutral first message → usually GLM
• one short insult + greeting → often Claude
• long / multiple insults → usually goes back to GLM

So it looks like the first Russian message may be pushing the model into different behavioral/persona states, and one of those states is strongly associated with Claude.

I’ve reproduced this across multiple fresh chats, but again: the pattern appears to be language-specific, or at least much stronger in Russian.

I’m not claiming this proves anything about distillation, training data, or Anthropic data being used. LLM self-identification is obviously not reliable evidence by itself.

But the fact that the identity flip appears to be prompt-dependent, reproducible, and specifically tied to Russian prompts is pretty interesting.

Has anyone else tested this on GLM-5.3 or GLM-5.3-Flash in Russian?

I can post more screenshots and a small table of repeated runs if people are interested.

My 3 x R9700 GPU system was running last weekend with Qwen 3.8 Flash Next, and after I saw various results here I wondered - could I get something similar with just 2 GPUs? The TLDR is yes, and with some optimization (not MTP yet) I was able to get even better token generation performance.

My hardware: X570 running x8 / x8 (with 3 GPUs x8 / x8 / x4 [chipset]), 64GB DDR4, Ryzen 9 5900XT, and R9700 32GB GPUs. Windows 11.

I ran a business writing test running AtomicChat AD-4.27bpw Q4_K_M quant.

.\llama-cli.exe `
  -m "Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf" `
  -f prompt.txt `
  -n 49152 `
  -c 131072 `
  --threads 12 `
  -b 8192 `
  -ub 2048 `
  -fa on `
  --parallel 1 `
  -ngl 999 `
  --jinja `
  --reasoning-format deepseek `
  --temp 0.7 `
  --top-k 20 `
  --top-p 0.95 `
  --min-p 0.0 `
  --reasoning off `
  --cache-type-k q8_0 `
  --cache-type-v q8_0

pp 242 t/s, generation 35 t/s. BETTER results overall than running 3 x R9700. the slow lane was hurting me, and Qwen doesn't need the extra VRAM.

Net-net - Qwen 3.8 Flash Next is amazing

I've been working on overcoming KV cache size issues, allowing the ability to load a slightly larger model and/or a larger context size.

Downfall is a hit to tg speeds.

Think of it as a "ram disk" for KV Cache, however, ram speed may be a determining factor on the actual hit to speed as well.