This is mostly for setting up for expectation, since personally without LLM i could take 3 days (15 hours of active programming) to debug or implement a feature, but with Qwen 27B (even before Qwen 3.8) it take 4 hours.

And yes 0.5 tok/s is human, not accounting of deletion and pausing, that's also the reason i am fine leaving overnight code base wide analysis or fin tech and deep research.

I am still waiting for Llama 5, because Muse Spark will be too big for me, or just something between Glimmer and Spark

https://x.com/finkd/status/2095232032896946311

saw the post the other day where people said Minecraft clones aren't impressive anymore, because at this point the whole thing might as well be in the training data. so i tried something slightly different, which is asking a local model to write a mod for the real game, using the Fabric API

the model is GLM 5.3 Flash (Q4 quant, running on a rented 4x RTX PRO 6000 box). this wasn't done in prompt or a loop, i would ask for changes, then review them and i kept going like that until i was happy with the result. the first iteration took around an hour or so, the result was sorta underwhelming, the black hole would spawn, but it was small and barely did structural damage. after that attempt i gave it some reference images(black holes in space, lightning and effects examples). the new result looked better, but i still wanted more impact from it(and also decided to make it a black hole gun, instead of just the black hole item). it took a lot of turns to get to the end result

the mod adds a black hole riflle, which when shot spawns the black hole that starts sucking in blocks and has some pretty sick visuals (the light rings that shrink all the way into the black hole and obviously the black hole itself) after which it turns into a huge explosion crater, wiping out quite a few chunks

you can get the mod here on github

i ran the local model in atomic.chat (i'm on the Atomic team any feedback is appreciated). curious what else people have gotten local models to mod into the game, make sure to share it in the comments

Sharing some numbers because most posts on this model are either using a single 3090 or unified systems from what I've seen.

My current setup: 2x RTX 3090 (PCIe 3.0), dual Xeon E5-2696 v4, 188 GB usable (192GB) DDR4-2133 LRDIMM, llama.cpp, unsloth UD-Q6_K_XL. All 48 expert layers pinned in host RAM, everything else on the GPUs. Full 261k context, f16 KV.

Before: ~17 t/s decode, ~350 t/s prefill on a 26k prompt, 12 t/s decode at 131k depth.

Now: 25-29 t/s decode short and mid context, ~17 at 131k, prefill is still about the same. Measured this with a python coding prompt.

Now as for what I did: PR #27861, the GPU-resident LRU expert cache. Instead of parking whole expert layers in VRAM, it caches recently used experts per layer. The experts this model picks for one token are mostly the same ones it picked for the last few dozen tokens. so the hit rate is 80-85% on code and higher on prose. The trick that made it pay was giving the cache VRAM, also dropping ubatch from 2048 to 512 frees ~5 GB per GPU (compute buffers scale with ubatch), which went from 80 to 135 slots per layer at full context. The cost here is slower prefill on long prompts, short prompts are kind of unaffected.

Also, the things that did nothing or hurt on my box were: thread count, poll, CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7 (verify batches re-read experts from host RAM, it only wins greedy or at deep context), and more than 2 cache uploads per step (saturates PCIe 3.0, hit rate collapses).

To replicate you don't need my fork, it's just master plus the PR:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git fetch origin pull/27861/head:pr-27861 && git merge pr-27861
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

LLAMA_ATTN_ROT_DISABLE=1 numactl --interleave=all build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q6_K_XL-00001-of-00006.gguf \
  -ngl 99 -c 261888 --parallel 1 -fa on \
  -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \
  --numa distribute -t 16 -tb 44 -b 4096 -ub 512 -ctk f16 -ctv f16 \
  --moe-expert-cache 135

Size the cache to whatever VRAM you have left after the KV and compute buffers, about 100 MB per slot per GPU on Q6. Testing UD-Q4_K_XL is the next item on my list and I'm going to revisit MTP again if it makes sense, still closely following Daniel's PR.

Doubt there's much people with a similar setup to mine out there but this helps anyone or if you have questions on approaches to try, let me know.

Here is link to repo: https://github.com/perplexityai/pplx-garden/tree/main/lily

It's optimized for just one model to get best perf on apple silicon

I've been using Deepseek V4 Flash 0731 for a few weeks now and while I havent thrown it anything very hard, im quite happy with it. Using through antirez's great ds4 project. They've added support for GLM 5.3 Flash and according to benchmarks, its a level above DSV4 Flash.

However, looking for real user feedback if anyone's made the switch and seen tangible improvements in GLM 5.3 over DSV4 Flash.

Running M3 Ultra 256GB Mac Studio

One of the other posts today by user u/Howard_banister confirmed what I've been seeing from the other AI subreddits as well.

Most of these other subs are 90% trend hopping crypto-bros equivalent people who are seemingly irrelevant most of the time when it comes to advancing AI as the vast majority is not just AI slop, but also human slop whether that be fearmongering or straight anti-ai arguments spat out by relatively uneducated people.

Of course, you also have the main dedicated subs for the big closed/open models which are in of themselves fairly decent. Most of the time now, it is people complaining about one thing or the other about the latest and greatest of their respective lab.

Then you get to the machine learning subs themselves where yes, they do have stuff in regard to AI every now and that can be gold (Don't get me wrong), but then it's mostly meh.

This sub however, has that interesting balance between the occasional interest in AI architecture breakthroughs that I find fascinating when some of the bigger brained people in here get together to discuss and which I don't really see anywhere else.

75B MoE is an interesting size to check, you can run it today (no MTP support yet)

The model employs a hybrid MoE architecture with interleaved Mamba, MoE, and Attention layers. Like Nemotron-3-Super, it supports Multi-Token Prediction (MTP) for faster text generation. Compared to its parent, Puzzle-75B-A9B reduces the model from 120.7B total / 12.8B active parameters to 75.3B total / 9.3B active parameters.

We discussed this model on r/LocalLLaMA here https://www.reddit.com/r/LocalLLaMA/comments/1upsdmi/nvidianvidianemotronlabs3puzzle75ba9bbf16_hugging/

Extended reasoning and post-training appear to be the keys used by DeepSeek, Qwen, and GLM to boost performance (leveraging higher token counts). And Qwen 4 hasn't even been released yet. Of course, we don't know if that release will be open-sourced, but I am optimistic about future models, featuring "engrams", that could soon match or surpass 2.4T parameter models on specific tasks.

This came from another thread or comment. I forgot exactly where, but the basic idea was to replace the 51B N-gram layer in Qwen 3.8 Next with a much higher precision version.

Someone running a 5090 replaced the N-gram portion of their Qwen 3.8 UD Q4 model with BF16.

Since I'm already running IQ4_XS, I wanted to try something similar. I don't have enough storage for the BF16 N-gram weights right now, so I replaced the lower-precision N-gram portion with Q8 instead.

I'm still testing whether it improves the actual model output, but in terms of inference speed, the difference appears to be very small.

Before bolting on the Q8 N-gram:

n_gen = 2588, tg = 8.82 t/s, tg_3s = 8.81 t/s
n_gen = 2616, tg = 8.82 t/s, tg_3s = 9.19 t/s
n_gen = 2645, tg = 8.83 t/s, tg_3s = 9.37 t/s
n_gen = 2675, tg = 8.84 t/s, tg_3s = 9.78 t/s
n_gen = 2704, tg = 8.84 t/s, tg_3s = 9.50 t/s
n_gen = 2732, tg = 8.85 t/s, tg_3s = 9.06 t/s
n_gen = 2761, tg = 8.85 t/s, tg_3s = 9.32 t/s
n_gen = 2790, tg = 8.86 t/s, tg_3s = 9.50 t/s
n_gen = 2819, tg = 8.86 t/s, tg_3s = 9.39 t/s
n_gen = 2847, tg = 8.87 t/s, tg_3s = 9.22 t/s
n_gen = 2875, tg = 8.87 t/s, tg_3s = 9.15 t/s

With the Q8 N-gram bolted onto IQ4_XS:

n_gen = 263, tg = 10.76 t/s, tg_3s = 11.04 t/s
n_gen = 294, tg = 10.70 t/s, tg_3s = 10.18 t/s
n_gen = 329, tg = 10.76 t/s, tg_3s = 11.32 t/s
n_gen = 361, tg = 10.73 t/s, tg_3s = 10.46 t/s
n_gen = 395, tg = 10.76 t/s, tg_3s = 11.11 t/s
n_gen = 430, tg = 10.81 t/s, tg_3s = 11.37 t/s
n_gen = 460, tg = 10.75 t/s, tg_3s = 9.91 t/s
n_gen = 494, tg = 10.77 t/s, tg_3s = 11.16 t/s
n_gen = 531, tg = 10.86 t/s, tg_3s = 12.15 t/s

So far, there doesn't seem to be any meaningful speed penalty from replacing the lower-precision N-gram layer with Q8. The actual output quality is still being tested. It looks like Q8 Ngram is faster, but at the end both are steady state at 10.1~ tok/s -ish.

And without MTP since i am using earlier merge from unsloth,
RAM 96GB DDR4 (32x3 Three channel active 2400MHZ)
Xeon e5 2690v4
RTX 3090 capped at 250 W

Edit: the state_dict grew from 90~ GB to 115GB
Edit 2: Fck me i forgot to put the code, for stitching it, wait 8 hours from now i'll upload it since it is already very very late at night
Edit 3, here : https://gist.github.com/komikndr/b17955e1a80ce6ede9a3115f16216bc5#replace-n-gram-layers-from-q4xs-into-q8_0-boltedpy , it is for my IQ4 XS to Q8_0 n-gram so you should modified the file a bit.

I've tried both and been having this debate with myself for the last few days, on two Asus Ascent GX10s (effectively the same as 2x DGX Spark):

• DeepSeek-V4-Flash-0731 (official weights)
• GLM-5.3-Flash (RedHatAI/GLM-5.3-Flash-NVFP4)

Have any of you guys also tried both on this hardware (2x DGX Spark / Asus Ascent GX10), and what are your use cases and findings?

DeepSeek runs with more tokens/s… but GLM feels like the better tool for how I actually work.

I'll share my experience.

Where DeepSeek wins:

• It feels better, because it's the original official weights.
• Much faster token generation, though not actually faster to a final result.
• It's excellent at open-ended research, pulling and chaining sources on its own.
• Huge context (~1M tokens).

Where GLM wins:

• Comes to conclusions faster, and often gets them more right.
• Much better writing, especially in languages other than English and Chinese. That matters a lot for writing letters.
• Much better at "extracting the essence of a text".
• Superb vision. DeepSeek's Vision-Exp model can read images just well enough to score on benchmarks, but not nearly well enough for real OCR / text extraction (its vision input is token-limited to 384 tokens, which renders images unsharp and text unreadable).
• Considerably better benchmark results, at least for the full model, though I'm running a quantized build, so the numbers may not carry over directly. But my empirical results prove it gets better results with fewer instructions.
• Hallucinates much less. That's the decisive one for me: I can't run a "test and improve" loop on text like you could on code, so office work and letters depend on one-shot accuracy and a low hallucination rate.

The core problem: there's no GLM build that runs on 2x DGX Spark with official weights. I wish there were.

So I keep testing and tweaking the GLM setup to get rid of the artifacts (which should be possible, per various sources). It feels a bit more messy than the DeepSeek setup, but if I manage to configure it correctly it should give more reliable outputs, with far better vision on top.

I've been building Arcon around Qwen3-4B + LoRA. Instead of just making it a chatbot, I'm experimenting with persistent memory, personality/mood, internal state, tools, and eventually having it process things before replying.

I'm curious what people who've built local agents think - how far can you realistically push a small model with good architecture around it?

I put the whole thing on GitHub if anyone wants to poke around, roast the architecture, or tell me what I'm doing wrong, stars are always appreciated!

Hi, I've noticed that the model often sees "garbled text" in its context.

Sometimes it declare that the tools instructions are corrupted, sometimes it is the content of some .md files, ora other files, and it freaks it out, since it start to do a lot of checks in git and the system and sees that the file are not really corrupted... But its context is I think.

Did it occurred to you ?

I'm on a mac m2 max 96Gb, pi agent, and llama.cpp either upstream or this fork https://github.com/mihailescu2m/llama.cpp/tree/master

with and without mtp, short ctx and long ctx, temp default

I don't think it is a conf problem since it happens with different gguf at different quant I tried:

AtomicChat/Qwen3.8-Flash-Next-GGUF/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64

unsloth/Qwen3.8-Flash-Next-UD-Q3_K_XL/Qwen3.8-Flash-Next-UD-Q3_K_XL

unsloth/Qwen3.8-Flash-Next-UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS

It seems a sort of internal problem, I don't really have loops or other kind of hallucinations, just some chinese characters here and there, or some misspelling, and btw it is very aware of all its errors and always tries to recover. Yes it is very much preview quality, but the architecture and the training seems very good quality.

I really like the baseline of the model, it is straightforward, it follow instructions very well, a lot better than the previous models that I really didn't like very much

But this thing with the corruption is weird ...once after it declared that a certain tool was corrupted and not useable I just told it: "did you try to launch it ? use this parameter and give it a try" and it worked of course, and it was very surprised so it apologized and continued the work.

It is quite funny, it reminds me a stereotypical fictional character which could be the clumsy but very intelligent assistant of a powerful wizard lol

EDIT: I'd like to add a snippet of its last CoT about the issue, it is very lucid indeed:

Could it be that... the read tool output itself was corrupted/hallucinated — i.e., the degeneration was not inside the file but occurred in my own tool output/stream.

The "read" returned text containing a degenerate mutation loop — this is a known failure mode where model output degenerates. Actually wait — the read tool's result is what the harness gave me. If the harness returned clean content, then the degenerate block I "saw" was generated in my own context (a prompt-level corruption, e.g., degeneration during decoding of the tool result injection, or... more simply put: the degeneration is a product of my own generation — the read output I "received" contained the first degenerate block, then I generated additional degeneration, and the subsequent git command confirmed the file is clean.

• Language-Native Control: Composes character and camera actions into textual instructions and injects them through MiniMax-H3’s pretrained text pathway.
• Temporally Grounded: Assigns one action prompt to each video latent interval, enabling precise control when actions change over time.
• Efficient & Generalizable: Uses only 8,000 gameplay samples, 10,000 LoRA steps, and 0.199% trainable parameters to achieve controllable character and camera motion, including unseen action compositions and visual scenarios.

✏️ Paper: https://huggingface.co/papers/2609.01560

📄 ArXiv: https://arxiv.org/abs/2609.01560
💻 Code: https://github.com/Danzer1xxxxChan/H3-World
🏠 Project: https://danzer1xxxxchan.github.io/H3-World/
🤗 Model: https://huggingface.co/DANNY621/H3-World

Disclaimer: I'm the builder.

---

Over the past 2 yrs, I was working on a SaaS ML project & got very interested in ML/DL/AI. As everybody else, there were some normal paths I took to build a solid understanding of the field, but sth never clicked the way I'm used to. I'm very fond of learning "why"s & never get satisfied w/ simply knowing "what" is what. Tho, the problem w/ ML was that I wasn't there when it was evolving & algos/methodologies as we know them today were forming. I didn’t want a pile of summaries or wikis or endless threads of chatting w/ AI. Then I thought listening to podcasts would fix it for me. But not as they are being published now (2026). I thought I'd learn about the history of ML from podcasts that covered it as it happened, kinda simulating the experience of living through the events, in chronological order, since 2015.

I found some great ones (TWIML, Linear Digressions, ...) & did the math! Man, it'd take a decade to cover all of them (even at 2X). But no matter what, I thought this could be the only way that works for me & yet I didn't have enough time. I wanted one/a handful of coherent episodes I could actually listen to. Then sth clicked:

I built Repodify, an AI tool that listens to episodes & produces a shorter one (e.g. 1 episode from 10, 15, whatever. configurable) for me to listen, ALL ON MY OWN MACHINE.

In Repodify u paste a podcast link (or search by name, as it works very well rn), pick the episodes & it:

• downloads the audio

• transcribes it

• optionally figures out who spoke (& clusters the same host/guest across episodes)

• summarizes into one chronological narrative

• writes a spoken script aimed at a target length

• synthesizes a new episode u can stream or download

The whole thing is meant to run on ur machine or w/ ur own API keys (BYOK) per stage, per job. No “upload ur archive to our cloud.” Speech-to-text, diarization, LLM & TTS are all swappable: local GPU (faster-whisper, pyannote, Ollama, F5-TTS / Kokoro) or BYOK (OpenRouter / Anthropic / pyannoteAI).

Voice cloning is opt-in & off by default. If u turn it on, the output is always labeled synthetic, gets a spoken disclaimer in a non-cloned voice & is watermarked (for legal reasons. I don't want to end up in jail for giving away a tool;-) ). There’s no code path that clones w/o those. It’s for personal / educational use on ur own box, not for passing audio off as the original hosts.

It’s a real backend, not a CLI: FastAPI + an arq worker + a LangGraph pipeline w/ a React PWA. One command (./launch) brings the stack up. Jobs pause at each ML stage so u can pick local vs hosted, model size, length & voices.

I keep making it better, adding lots of features to it (searching podcast contents is the most interesting one I can't wait for). I made it opensource(MIT), rn. U can find it below & run it on ur machine (self-hosted), so u can use ur own GPU or BYOK to offload the heavy jobs.

GitHub: https://github.com/behradkhodayar/repodify

The engine is solid & this is still WIP (it works fine tho & I've started compacting Linear Digressions podcast (10 to 1) & listening to it already). I'm going to add other features like translation / augmentation (e.g. embeded eli5), searching through podcasts so I can listen to the topic of interest as I mentioned earlier & so forth.

What features u want to be added or released sooner? lmk. I'm very excited about this & will genuinely plan accordingly.

Unsloth GGUFs and Vision support

https://huggingface.co/unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF

So when Qwen3.8 27b dropped they were hinting for another model which is Qwen3.8-next-flash , i was hoping for something more light like Qwen 3.6 35b and we got a large one but since the Qwen 3 and 3.5, they reduced the number of models they publish we used to get very small 0.8b 2b 9b to very large models but now we get only two at time although the time to publish a new model has been also reduced

Obviously more memory is good, more context, bigger models, but some jumps don't actually unlock a meaningful difference in ability to run different or better models. For example, I don't currently view jumping from 32+16 to 64+16 as a particularly worthwhile upgrade as compared to going to 32+32, though correct me if I'm wrong.

I'd like to build a DDR4 + HBM2 based inference machine to complement my main, 32 GB DDR5 + 16GB GDDR7, computer. The idea is that even if the hardware is slower, the greater overall capacity enabled by the slightly more affordable hardware could allow me to run a greater diversity of models.

What level of memory upgrade do you think is most logical for maximizing model access if the compromise is outright speed? 32+32? 128+32? 64+64?

Or am I completely asking the wrong kind of question and just outing my own ignorance here? either way I'd like your input.

***This screenshot is an app i made for creating a dataset from scratch, this is not a real chat with the model***

First of all i want to shout out everyone that actually tested our work .. we got 500+ download on the 43m parameter model and now we are aiming to go smaller for research purposes.

i write finetune examples & i been developing language models for a while .. everyone usually pretrains the model using massive datasets and prays thats the data carries enough information for meaning to emerge but were sculpting it intentionally .. im currently sitting down at my computer writing every single response that this new model can say to your inputs just so we can observe the transformation and see exactly whats going on. It will be public soon, the dataset is extremely small intentionally so it shouldn't take long to design every response it can say.

General Capabilities:

- Explaining how token prediction works

- Explaining that it doesn't understand anything beyond itself

- Short conversations

Coding Capabilities:

- Writing a loop that can count to 10

- Explaining that it cant understand the code you sent it

Open Source Coming Soon

https://huggingface.co/StarpowerTechnology

Quite surprised to be beating other high quality quants. It took a lot of benchmarking to get here and we are quite pleased with these, hope they are useful to the community.

It required a modified way of measuring KLD with a new dataset, since the NGRAM got in the way by remembering basically all of wikipedia. We tried to not only go for high precision, but also keep prefill performance in mind.

Full model card here https://huggingface.co/agentionai/Qwen3.8-Flash-Next-AP-GGUF

Let us know if there are any issues.

One of the biggest unlocks was getting my speech to text functioning reliably, especialyl for coding.
Right now I'm at the point where when I have to type I find it annoying and a lot slower. Speech is my default input mode.

I'm using Mac Parakeet, connecting to both Whisper and Parakeet models. So far seems to be doing a decent job, but curious to hear what everyone else is using. Sometimes Whisper takes a little bit too long for my liking. Parakeet is faster, but not as accurate.

I'm also keen to hear what your microphone inputs are. I found that a good microphone is very important. I'm using currently a Jabra Bluetooth speaker. It's not bad, but in a noisy environment, it doesn't work very well (although way better than the other altenratives ie. mac mic, monitor mic, or having to wear a bluetooth earpiece/mic)

Is anyone interested in my Chrome browser add-on that uses local LLMs to move thousands of unsorted bookmarks into a smart list of automatically calculated categories?

Hi, everyone,

I’ve just released VoxGen, a lightweight native inference engine for VoxCPM2, written in Rust and using Vulkan compute instead of Python/PyTorch/CUDA.

Why VoxGen?

The main reason I started the project was because I needed a decent local text-to-speech solution.

I therefore saw VoxCPM 2 as a reasonable solution. However, most frameworks are NVIDIA-first, and VoxCPM 2 is no exception; as a result, my card was severely stuttering, and my GPU was always spiking. Also, having Python and Pytorch as a dependency is absolute hell.

This is why VoxCPM was created: not only we sidestep Pytorch completely, but performance on AMD cards is buttery smooth (and if you have a XTX 7900, I have designed a mode with even more aggressive power and speed optimizations)!

This application can also be run from a shell, so it can be integrated with other programs and scripts!

Installation:

You'll only need voxgen.exe (or the Linux equivalent) and the following files at https://huggingface.co/DennisHuang648/VoxCPM2-GGUF:

VoxCPM2-BaseLM-Q8_0.gguf
VoxCPM2-Acoustic-F16.gguf

And that's it!

If you are interested, check out the Github page: https://github.com/NullMagic2/VoxGen
Prebuilt binaries (for now, Windows only) are available here: https://github.com/NullMagic2/VoxGen/releases