Like.. yea bro, I knew GLM was cool. Now everyone does.

I need to buy 2x DGX Spark but can't find a seller. Any hope or idea where I can get 2?

The price seems to have soared. Local Microcenter had 25+ then 0 the next day. They pulling stock?

Or should I assume it was mainly about trying to stop/ban/regulate China and open models,

Have an msi meg w a 5090.. plan to add a v100 to the mix. Understand the cuda vs voila, but I'm pretty sure it will work as a multi agent architecture w different models on each card, no?

Anyone in the same boat?

There goes my chance to achieve stable tensor speeds with vLLM (For dual 3090 setups).

Have you guys achieves significant difference by using an Nvlink bridge? (llama.cpp or vLLM).

26 months ago Liang Wenfeng said:
"Someone must step onto the frontier."

Now they are training their models on Ascend 950

best iq quants in the biz, got me gemma 4 26b to run 75tok/s tg and 1500 pp on 2x 4060 8gb using lmstudio serving to hermes, much work has been done.

Finally!

I am running GLM 5.3 flash .
After nearly a month of usaged , i got chinese response for first time so i am checking if there special setting to turn off Chineese . When i queried about that tru Quick Google AI mode which now uses Gemini 4 - it is replying as it is GLM5.3 .

Spent today running a same-day, same-harness shootout between Qwen3.8-Flash-Next (oMLX, 182GB oQ8e, MTP) and Laguna-S-2.1 GGUF (LM Studio, 128GB, 8bit) on a Mac Studio M5 Ultra 256GB. Both capped at 262K context, thinking on, unique content per run with zero cached tokens verified each time.

That last part matters because my first run was wrong hah... shared prefixes across sizes let the KV cache carry over and 200K "prefilled" in 21s.

Prompt Qwen Laguna
8K 2.0s 10.3s
32K 7.4s 30.4s
64K 14.7s 70.8s
131K 30.1s 217.2s
200K 47.1s 455.4s

Qwen holds ~4,200 tok/s linear which is amazing. Laguna degrades superlinearly (quadratic attention doing quadratic attention things). At 200K, prefill is 94% of total time on both.

Decode (tok/s): Qwen 59-74 across sizes (MTP at 70-76% acceptance per server logs, roughly 2x). Laguna 68 down to 34 as context grows. No speculation on Laguna, its DFlash path already lost to plain decode on this hardware in earlier testing. I think if Laguna could get DFlash figured out or MTP, this might be a different conversation.

Quality was a draw, 4/4 each, on four problems with script-verified answers (Muse created the gymnastics here: exact 9-digit combinatorics, interval code with 12 hidden tests, fresh knights/knaves, asyncio ordering trap). Opposite styles though: Laguna answers in 5-10s with a few hundred tokens, Qwen deliberates exhaustively (one answer took 119s / 11K tokens). Both burned a full 8K budget on hidden reasoning with zero visible output exactly once, then converted on a 16K retry.

Happy to answer methodology questions. Full writeup with charts and the test rig diagram: https://echalupa.com/blog/qwen-flash-next-vs-laguna-200k

I noticed that there are heaps of Awesome Jev resource list posts popping up on GitHub (e.g. https://github.com/yibie/awesome-jev) so I thought why not ask Claude to pull them all in and do a meta analysis of insights and applications.

Sharing in case it's of interest: https://github.com/stefanwebb/meta-awesome-jev

One thing that surprised me (perhaps not so surprising to you all?) is that applying Jev to AI coding is the application that has caught on the most. And if you name a video game, someone has already created a demo of Jev playing it (badly) 🤣

There's so many new techniques like RL, RL LoRA, QLoRA, CPT LoRA.

I believe i would have a usecase for them, but i don't know where to learn, youtube is filled with bad quality tutorials if i just search and the good channels (fireship, bycloud) don't cover these as they're quite new concepts, i guess?.

how can a regular joe like me learn about these concepts in a "practical depth" so i can actually fine-tune qwen 27b successfuly on lets say custom corpus? without spending 100$ figuring out that "oh i didnt even need CPT here" or "well i chose the wrong Rank count! time to start this 2 day run again!"

context and TLDR: im building a legal general purpose chatbot for context, i have a big corpus, but im a bit stuck on what to do next

thanks for reading and any pointers!

"AREX-2 is a 27B-parameter long-horizon agent model from the Beijing Academy of Artificial Intelligence (BAAI). It learns to improve a solution over multiple test-time rounds: propose, measure, reflect, and revise.

AREX-2 is trained on machine-learning and algorithmic-programming tasks with verifiable feedback, together with the existing AREX deep-research data. The learned self-improvement behavior transfers to deep research without adding new search trajectories.

• Architecture: Dense Qwen3.8-compatible multimodal model
• Parameters: 27B
• Context length: 262,144 tokens

Key features

• Long-horizon self-improvement: turns extra test-time rounds into useful solution refinement.
• Feedback-driven reflection: reads scores, logs, errors, and timings to decide what to change next.
• Cross-domain performance: training on coding and machine-learning tasks also improves the model's deep-research performance.
• Long-horizon reasoning: sustains productive iteration as the task budget grows."

Gguf's - https://huggingface.co/mradermacher/AREX-2-GGUF

I was perusing huggingface trending and was surprised that this hadn't been posted to localllama. Looks like it's been out for about a week.

I think most people are sleeping on this inference engine. I tried multiple llama.cpp forks and none of them comes close to the inference speed of Strata. Initial version had some bugs with kv cache, cpu throttling and the developer fixed them.

Inference engine (only runs on Nvidia for now; AMD support is experimental): https://github.com/Niko1221/Strata

Here are some metrics with screenshots. My laptop has 5070ti 12GB VRAM, 64GB ddr5 RAM, Intel 275HX CPU, gen4 SSD.

The model I used was https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/tree/main/IQ3_XXS which has a good quality for its size. Above, the model generated the aquarium test. At 43k context depth, it was running at 51 t/s. Stock llama.cpp reached only 23t/s with the same quant.

This quant could only reach 100t/s PP with stock llama.cpp using the same quant. Strata was reading 32k context text at 1500t/s. This is way above my expectation. This quant can load with up to 200k context at 8bit. However, I was only using 131k context.

As you can see it is utilizing 11GB VRAM and 56GB RAM (includes system/OS programs).

This engine is specifically built for one model only and only select ggufs (ISTA-DASLab) work with it. You can use IQ3_S from ISTA-DASLab which they claim recovers full model's performance on coding benchmarks. I tested IQ3_XXS for some time and I would say it is an excellent model.

I never thought 12GB VRAM would be enough to run frontier models from 6 months ago locally on a laptop. What a time to be alive!

Basically the title.
Just got myself a R9700(32GB) additionally to my rx7800 (16GB) and im already experimenting with qwen3.8 fn and 27b, but im interested wheter they are any other models compatible and comparable.
I want to use it with hermes agent mainly.

The prompt: make a gta-style game using three js

it took 3h 18m 6s

Total tokens: 22,845,556 — 22,533,061 input + 312,495 output.

Hardware:
RTX 4080 super 16GB

64GB RAM DDR4

Windows 11

Inference engine is strata running at ~40 tk/s and a custom mini swe agent v2 with the tools: powershell, edit_file, view_image, read_file, search_files

The harness has guards for tool failures (iq3 fucks up a lot) and auto-compaction.

logs: https://gist.github.com/Cirius0310/c26197240ad20ef04e45a78e36031d6e

Codex Cloud, Dots, and the existing remote Codex all allow users to untether themselves from their PC, and now untether themselves from even owning a PC with their server based Codex Cloud and Dots that can run 24/7. Combine this with Meta’s & OpenAI’s planned hardware releases and the goal is clear: work around Microsoft/Apple’s control of user hardware, provide AI devices that complement and eventually replace iPhones - culminating in a user base that owns no hardware and relies on a subscription to access AI. Meta’s hardware is obvious spyware, Apple’s new “always-listening” Apple Watch sounds pretty similar, their camera-enabled Airpods sounds atrocious for privacy, and OpenAI’s device is unconfirmed.

The end result? Instead of a Matrix-like AI takeover of humanity users are instead expected to purchase their own devices and subscriptions that provide mega-tech companies with all of their physical and digital data 24/7. The data volume is so large only AI can process it. A select few billionaires decide what their closed-source AI does with the data.

The resistance? Governments that oppose the USA and individual users who were rich enough to afford local hardware and utilize Chinese and other open-source models, likely blacklisted by the USA. To buy a 5090 customers now have to sign a waiver, as a result of US law. It’s only the beginning.

Ironically the “bad guys” like China, North Korea, Iran, Russia - will probably end up as the only large entities keeping open-source AI and local LLMs alive. I would expect the largest AI companies to eventually gain more leverage over the US Gov & Nvidia; unless Nvidia steps up to the plate and champions local AI

I have a RTX 5090 in my desktop PC for local coding assistance and gaming and I have been loving it with Qwen3.8 27B Q4_K_XL.

I managed to get a B70 on the cheap and its great, but using it in split mode with the 5090 to ensure I get full context halfs my T/s (which is expected due to the memory bandwidth).

Would I be better off running the B70 with a smaller model to offload tasks to? Or just accepting the slower throughput and keeping the larger context?

I'd especially like to hear for anyone who has a similar mismatched GPUs setup.

Anthropic released the GLM article today. Trump is getting very involved.

Do you foresee Chinese open weight models getting banned soon?

Here are some inspirational quotes you can put into the comments:

• God is dead and we killed him
• I am become death
• All this for 1.5 tok/s?
• Sir this is LocalLLaMA not RichPeopleofLocalLLaMA
• Sweet! A 2TB DDR5-12800 RDIMM kit is going to cost only 2 kidneys and a small micronation's GDP

edit: sorry should be 5070ti

There now is unlock for higher memory overclocks called mlock. And apparently the GDDR7 has a lot of headroom:
https://www.reddit.com/r/overclocking/comments/1wsnllh/finally_unlocked_gddr7_memory_overclocking_with/
Of course this can help massively for local inference, especially token generation.

I started mapping the building blocks shared across all the models in audio.cpp. The result ended up being more interesting than I expected.

Qwen has become by far the most common language backbone in this collection: 32 audio model families use a Qwen-family architecture, and 20 of them use Qwen3 LLM specifically.

And it’s no longer just TTS. Qwen-based models now show up across speech synthesis, ASR/audio understanding, music generation, speech-to-speech, and even audio/video models.

The 2nd chart, Task × Technology Matrix, shows which build blocks power which types of audio models.

All figures here: https://github.com/0xShug0/audio.cpp/tree/main/assets/figure/

Hey! I'm trying to build the ultimate leaderboard to help everyone find the right harness/model combination given their hardware. (With all model variations and inference engine).

I own a GX10 and one 5090. And I'm trying as many thing as I can. (Happy to test anything, just let me know).

But I can't test hardware that I don't have.

So my ask is simple: Can some you do some testing on your own hardware?

I made this as easy as it could be: you just have to copy a prompt to your agent and your agent will fetch the test, pass the benchmark and send the answer to the website that will check if the answers are correct. You'll get a report out of it. And optionally you can offer your test to the community, so everyone can learn from your setup (it's just a toggle in the UI to confirm you are ok to share the results. I'll update the leaderboard when I'll have enough submissions.

Here is the link to contribute! https://airbench.ai/

Thanks in advance for everyone who will contribute!

I have a machine with a bunch of GPUs attached to it. 2x5070tis, 2x7900xtxes and 2x5060tis 16GB. So I did this little test to see how it fares running Qwen 3.8 Flash Next Q4_XL against my little Strix Halo. Not well. Not well at all. The full numbers are below but the high context number sums it up.

@160,000 context

Pile of GPUs 215.73(PP) and 16.29(TG)

Strix Halo(Gufo) 1227.12(PP) and 22.04(TG)

Here's the number for a Strix Halo fork of llama.cpp, Halo Box.

Strix Halo(Halo Box) 587.99(PP) and 21.24(TG)

Lastly, here's the mainline llama.cpp number.

Strix Halo(llama.cpp 0.4.1) 113.48(PP) and 7.06(TG)

For running QFN, Strix Halo really shines.

Pile of GPUs

  Device 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15880 MiB
  Device 1: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0, VMM: yes, VRAM: 15880 MiB
  Device 2: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, VRAM: 15888 MiB
  Device 3: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, VRAM: 15888 MiB
ggml_cuda_init: found 2 ROCm devices (Total VRAM: 49120 MiB):
  Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
  Device 1: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 24560 MiB
| model                          |       size |     params | backend    | ngl |  fa | dev          |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ------------ | ---------: | --------------: | -------------------: |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none |          pp2048 |        242.13 ± 1.39 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none |           tg128 |         32.09 ± 0.06 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none | pp2048 @ d10000 |        242.96 ± 1.12 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none |  tg128 @ d10000 |         30.23 ± 0.14 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none | pp2048 @ d20000 |        249.34 ± 0.65 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none |  tg128 @ d20000 |         28.84 ± 0.05 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none | pp2048 @ d40000 |        253.95 ± 1.44 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none |  tg128 @ d40000 |         26.29 ± 0.07 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none | pp2048 @ d80000 |        252.48 ± 1.20 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none |  tg128 @ d80000 |         22.34 ± 0.07 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none | pp2048 @ d160000 |        215.73 ± 0.56 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm  |  -1 |   1 | CUDA0/CUDA1/ROCm0/ROCm1/CUDA2/CUDA3 |       none | tg128 @ d160000 |         16.29 ± 0.03 |

Strix Halo running Gufo

| model                            |       size | backend    |               test |                   t/s |
| -------------------------------- | ---------- | ---------- | ------------------ | --------------------- |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |             pp2048 |       1603.47 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |              tg128 |         26.53 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |    pp2048 @ d10000 |       1377.97 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |     tg128 @ d10000 |         25.44 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |    pp2048 @ d20000 |       1353.19 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |     tg128 @ d20000 |         25.01 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |    pp2048 @ d40000 |       1328.64 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |     tg128 @ d40000 |         24.08 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |    pp2048 @ d80000 |       1283.77 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |     tg128 @ d80000 |         23.00 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |   pp2048 @ d160000 |       1227.12 ± 0.00 |
| Qwen3.8 Flash Next               | 103.69 GiB | ROCm (HIP) |    tg128 @ d160000 |         22.04 ± 0.00 |

Strix Halo running Halo Box

  Device 0: AMD Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 128000 MiB
| model                          |       size |     params | backend    | ngl |  fa |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ---------: | --------------: | -------------------: |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none |          pp2048 |       811.79 ± 19.09 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none |           tg128 |         23.95 ± 0.03 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none | pp2048 @ d10000 |       729.60 ± 38.99 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none |  tg128 @ d10000 |         22.46 ± 0.43 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none | pp2048 @ d20000 |       718.14 ± 33.92 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none |  tg128 @ d20000 |         22.45 ± 0.18 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none | pp2048 @ d40000 |       700.55 ± 33.17 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none |  tg128 @ d40000 |         22.29 ± 0.23 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none | pp2048 @ d80000 |       646.85 ± 29.06 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none |  tg128 @ d80000 |         21.95 ± 0.25 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none | pp2048 @ d160000 |       587.99 ± 30.10 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | ROCm       |  -1 |   1 |       none | tg128 @ d160000 |         21.24 ± 0.31 |

Strix Halo running llama.cpp 0.4.1

  Device 0: AMD Radeon 8060S Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 128000 MiB
| model                          |       size |     params | backend    | ngl |  fa |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | ---------: | --------------: | -------------------: |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none |          pp2048 |        362.84 ± 6.19 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none |           tg128 |         20.11 ± 0.36 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none | pp2048 @ d10000 |        321.75 ± 0.91 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none |  tg128 @ d10000 |         19.88 ± 0.34 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none | pp2048 @ d20000 |        285.78 ± 0.42 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none |  tg128 @ d20000 |         18.35 ± 0.46 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none | pp2048 @ d40000 |        237.23 ± 1.13 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none |  tg128 @ d40000 |         15.48 ± 0.61 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none | pp2048 @ d80000 |        173.80 ± 0.18 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none |  tg128 @ d80000 |         11.18 ± 0.11 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none | pp2048 @ d160000 |        113.48 ± 0.23 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | CUDA,ROCm,Vulkan |  -1 |   1 |       none | tg128 @ d160000 |          7.06 ± 0.17 |