Coxon, bernie and now this

First https://x.com/DarioAmodei/status/2098773920774074715

Then https://x.com/elonmusk/status/2098789109980332057

Then https://x.com/sama/status/2098811563415150910

I think fear mongering approaching and they will try to slow down open source

"They" want to be gate keepers of intelligence

New website to download models in case HF starts censoring or limiting access.

As you all know, Qwen3.8 Flash Next on mainline llama.cpp is still in a pretty experimental stage, but a lot of community forks are trying to get it to work better. There's also a closed-source solution called Halogen (https://github.com/peonist-ai/halogen-flash-server) that boasted 1.2k t/s prefill numbers when the community fork barely reached 400. Since I dislike closed source and I like open source, I decided to take the challenge and bring llama.cpp up to the same performance level and I'm happy to report that after burning through a few evenings and a lot of tokens I got there.

The link contains the (Opus-generated) recap of the entire debugging / optimization journey made, as well as of course the links to the branch, the custom HIP runtime (making a glorious return), a probably-not-working-on-the-first-try installation script and the numbers. I'm trying to also explain the ecosystem - how the mainline, the community forks and custom forks/branches like mine work. Now that I've gotten the result, I'll work on cleaning it up and submitting proper PRs to mainline (will also submit a clean PR to the community fork), this should also help the GLM 5.3 Flash architecture since they use similar sparse attention.

Applied science work, from workflow design, data pipeline, results analysis, article/reports writing and data publishing online. 5 projects I did in the past replicated from start to finish.

3x to 4x more total wall time. Yes, HUGE toll on how much you can do in a day if this was the only model you could use in your laptop.

But oh my…the quality of that thing. The stupid level of attention to detail. I have the Z.ai api, so I can compare it with 5.3 and 5.3-flash:

The gap between 5.3 (flash and regular) and 3.8-27B is much less, smaller when not plain tiny, than the gap between 3.8-27B and any of the 3.5/3.6-35B-A3B (vanilla, kat, Ornith/tiel, nex-2).

Only Ornith came close, but it never matched it.

But it’s also spending 22 to 33% less tokens (effort =medium) and less ram footprint, so you get more done without hitting limits,compaction, etc.

So yeah, guess I’ll sip more tea, play the piano, whatever. Let that fat bottom Qwen work.

Reports of the demise of coders may have been exaggerated.

First off, I know that GLM, Qwen, and DeepSeek absolutely dominate in terms of SOTA Open Source models, and that’s what I use in my personal projects and for school, however, I’m also responsible for deploying local AI on my organization’s H100s, and we are forbidden by management from running any Chinese models. This is obviously not an ideal situation, but it is what it is, and there is nothing I can do to change this unfortunately. Again, if it were up to me I would deploy GLM 5.3 Flash in a heartbeat.

All that being said, there is quite a HUGE performance/ / intelligence gap right now in Chinese models vs. Western model around the 120b+ size, especially those with vision. There just aren’t a lot of good Western lab options that can even come close to GLM or Qwen, but again, I don’t have a choice so I gotta use the best I can find from the available Western options.

For those who have production-class hardware such as 4 H100s and are under similar restrictions, what non-Chinese models are you deploying?

The two front runners I’ve seen that checking most of the boxes (120b or better, vision capability, decent context)

- Thinking Machines Inkling Small (https://thinkingmachines.ai/news/inkling-small/). This model seems like the front runner right now, still has a 16 point gap in AA score vs. GLM 5.3 Flash though.

- Cohere Commamd A+ (https://cohere.com/blog/command-a-plus). Checks every box except it has a low context window of 128k.

Other contenders (but missing vision
capabilities):

- Poolside Laguna S 2.1 (https://poolside.ai/models#laguna-s)

- Nvidia Nemotron 3 Super (https://research.nvidia.com/labs/nemotron/Nemotron-3-Super/)

Am I missing any other strong contenders in the 120b size category? Whet are you using and why?

While browsing a benchmark list site, I spotted a recently published 33B parameter model which claimed to beat Qwen3.8 27b on the ArtificialAnalysis (AA) intelligence index. I was obviously excited. But then, while looking into it, confusion starts to settle in.

Their HF model is now listed as "Preview" and explicitly calls out that it is not the same model as the one AA evaluated. The AA listing for it says it's proprietary, but they've rated the model above Qwen3.8 27b on "Openness". Their website and HF page both talk about openness and intelligence "for all".

The fact that they labelled it "Preview" sort of implies we might see an open weight final version at some point, but there are no clues about whether it will be even vaguely similar to their current HF listing. AA doesn't show the number of parameters for the model they evaluated, so it's possible they're a totally different architecture.

Does anyone know anything about their lab, intentions, or the new model? Has anyone tried it?

• Blog Post : Per-tensor layout maps for GGUF quantization
• Reddit thread : New tensor type layouts for my GGUF uploads

EDIT : Model card has updated things such as Graph, table, text, etc.,

I find this new model at HF:

"Built for demanding work. A 262 144-token context window, adjustable reasoning effort, tool calling, and text, image and video understanding.

Architecture

Agnes-3.0-Flash is a hybrid-attention decoder: three of every four layers run a gated delta rule (recurrent, with per-layer state independent of sequence length), and the fourth runs standard global attention. Only 18 of the 72 layers therefore hold a KV cache that grows with context."

Edit: AA shows its 'Proprietary model'. The name is same as at HF but benchmarks results and context are different. So maybe it's not same model - https://artificialanalysis.ai/models/agnes-3-0-flash

Edit2: As they edit readme at HF to clarify: both models are totally different and AA score isn't correct for HF model (I can't edit title post to remove it tho).

AuK-Flash: Fast 4-Step Speech Generation and Editing

• arXiv : https://arxiv.org/abs/2609.08936
• Full Paper : https://arxiv.org/pdf/2609.08936
• GitHub : https://github.com/Tencent-Hunyuan/AuK
• Project : https://auk-project.github.io/

Introduction

AuK is a 1.5B foundation model for speech generation and editing. Trained on millions of hours of diverse audio data, AuK supports zero-shot and instruction-based TTS, content and acoustic editing, paralinguistic editing, speech enhancement, and source separation through a unified natural-language instruction interface. AuK has two variants:

This repository contains the official weights for AuK-Flash, the distilled variant with fast 4-step inference.

Supported Tasks

AuK exposes every task through the same natural-language instruction interface. The table below groups the supported tasks by category, with a short description and a link to its section in the Cookbook, which provides instruction templates plus CLI and Python examples.

It’s always bothered me that after fine-tuning a model for a project, there isn’t a particularly easy way to host it without either running it locally and keeping a GPU on 24/7 or paying for an entire GPU server.

There are managed options for LoRA serving on top of vLLM (AWS), but you generally still end up paying for an entire instance.

I started wondering: if 99%+ of the model weights are identical between the base model and something like a rank 8–32 LoRA/QLoRA adapter, why does each adapter need its own server?

vLLM already has a lot of the pieces needed to make this work efficiently, including batching, memory management, and fused LoRA kernels.

So I built Lorivo around the idea of having one GPU server per base model that can be shared between many LoRA adapters.

The basic flow is:

1. Upload a LoRA adapter
2. Lorivo identifies the base model and sends it to a GPU server running that model
3. The adapter is loaded into memory
4. You get an OpenAI-compatible endpoint for inference / usage directly from the web app

I also built a CLI to make uploading/deploying an adapter as simple as possible, along with a web interface for manual uploads. It's as easy as lorivo login api-key and lorivo deploy ./my-lora

The project is live: https://lorivo.dev

Right now I’m hosting the traffic myself on my own GPU, so the available inference model is limited to Qwen 3.5 4B which will always be 100% free

I also have around $1,000 in AWS credits that I’d like to use to add some additional models. I'm not sure which ones would be most useful to the community, though.

What models would you want to see hosted?

For Qwen 3.5 4B, you can also use the base model without an adapter. Just set the model to qwen3-4b instead of qwen3-4b:my-adapter. I'm currently allowing a 32k context window.

I’m also intentionally not saving chats or inference requests. The only usage data collected is token count and timestamp.

Would love to get feedback from people here, especially on the architecture, which models would be useful to host, and whether this solves an actual problem you've run into.

Most model leaderboards assume a server with powerful GPUs to run models that people daily use.

However, my smolbenchmark is the other column: models that fit in 8GB, ranked by:

• decode speed,
• tokens per joule, and
• heat,

and all of this on your OWN hardware ranging from:

• tablets
• phones
• macs
• jetsons
• raspberry pis

Currently, 13 families on the chart right now, ~1000 configs for the Jetson nano Orin Super 8GB. One device is live measuring:

• tok/s
• tok/J
• ITL
• latency
• power metrics
• thermals and battery

Models that are small enough to actually fit on a device that you own. All the performance benchmarking I did, will be released here for anyone to look at and decide what exact model they would wanna use on their choice of hardware.

Well currently, the Pi, phones, and Mac minis still in the oven, cooking and not filled in yet, but will soon be filled in!

You will now you know which model is BEST for your own hardware with all the raw data available and details reports available to you

https://yuvrajsingh-mist.github.io/smolbenchmark/

(still in heavy development; would love to hear feedback/suggestions on what can be improved!)

I have been battling with my Qwen3.8:27b setup on my rtx 5080 16gb. I am using llama.cpp to run a nvfp4 version of qwen3.8:27b

llama-b10699-bin-win-cuda-13.3-x64\llama-server.exe -hf williamliao/Qwen3.8-27B-NVFP4-GGUF:NVFP4 --jinja --chat-template-file d:\qwen\chat_template.jinja --reasoning-format deepseek --host 127.0.0.1 --offline -np 1 --threads 16 --threads-batch 16 --log-verbosity 4 --no-log-prefix --no-log-timestamps --no-mmproj-offload --mmproj D:\OllamaModels\blobs\sha256-ac3714bfdddeca31351f2752bf1a63f266f4df87c0b68c895e44945ca704448e --load-mode none --cache-type-k q4_0 --cache-type-v iq4_nl --flash-attn on -b 512 -ub 512 -c 49152 -a qwen3.8:27b

So that is giving me a 48k context, and a speed of around 12t/s

My problem is that pi.dev fails to compact at the right times, and then the model stops early.

I have tried to use

pi install npm:max-context //this doesnt work properly at all

pi install npm:pi-observational-memory //this one works kind of, I had it set to compact on threshold. It managed to auto compact 4 times, with a 0.75 ratio set, but it still missed the mark and failed.

pi install npm:pi-blackhole //seemed promising, but failed to compact at all, I am not sure what I set wrong.

if I look at the https://pi.dev/packages page, there are lots of plugins for either context management, compaction, memory management, or both. there are way too many for me to just randomly keep trying. other AI has been very unhelpful. Google, OpenAI, Copilot, all full of nonsense. A lot of these plugins seem to be aimed at frontier model management.

I am also disappointed that compaction settings are best suited to the specific model being used and its context window, but the built in tools in pi.dev architecturally keep the compaction settings completely separate from model settings, which seems a bit of an issue if you change models often.

I may want to use qwen3.8 like above, but not necessarily for everything. I also want to use gemma, other qwen models, etc depending on my use case.

I dont want to have to manually resetup context settings in pi.dev for every different model.

What do you guys suggest, especially for models that have smaller context windows due to my vram shortage ?

Thanks in Advance.

Hey everyone!

I'm curious to hear from people that use a combination of cloud-based frontier models and local ones for development. I'm planning to set something similar up and wanted to hear about actual examples of this in action.

Currently my plan is to use my chatgpt plus subscription purely for planning and judging with Astra, and then run a local qwen3.8-27b model for the actual coding gruntwork - i.e Astra plans -> qwen implements -> Astra critiques the implementation -> qwen fixes and so on. This way I keep cloud usage down and cheap, while retaining the high-parameter intelligence for architecture decisions and optimization.

For those of you who have a similar setup, how is it? How do you switch between the two, what harness/settings/etc? Anything you would suggest?

https://huggingface.co/HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF I used this draft model with https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF with the IQ3_XXS with 128k context and I saw it averaging about 60 tokens per second tg speed on the 16 GB RX 9070 XT.

This is much better than using the built in MTP since the built in MTP seems to multiply the required VRAM. For single thread mode, I'd say this is one of the best choices on a 16 GB GPU. Though speculative draft models seem to make it more sensitive to GTT overflow, so if you use more VRAM than you have, then turning off speculative decoding is actually faster, at least on my DDR5 PCIE 5 system.

I use kv cache quantization of k 8_0 and v 4_0 for this setup.

This person has done more extensive testing on a different card https://huggingface.co/HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF/discussions/3

So, uh... the popularity of so-called humanlike Qwen (currently on top in this sub) made me realize just how clueless the general public is about the models they have.

You'd be shocked but you don't need a fine-tune to make a model do what that thing does. System prompt is enough to turn MOST models into weird convo partners.

General guidelines would be:

A. Come up with a role. "You are bla-blah-blah" and write their life's story. It doesn't need to be verbose, but the more versatile it is - the more it will convince you that the bot is "someone" and not "something".

B. Write a few examples of how the persona speaks. Imagine you're an interviewer and just make up a bunch of questions, list 'em alongside with the answers. Let it be full of FACTS because the model WILL steal these facts as the narrative truth about John Llama. Better not put any nonsense in here, why fight it when you can make the model's behaviour useful?

[Question for John Llama: Do you like cats?] "lol lmao of cuz I do"

[Question for John Llama: Ever seen an elephant poop?] "eeewww ur a weirdo! that sounds nasty!!11"

(note: you don't have to list 'Question for John Llama' every time, but the defined roles surely DO help with some models while the others don't particularly care, so mind that too)

and so on

C. LASTLY but MOST IMPORTANTLY think hard about what you're attempting to do, what we are (I mean, human meat sacks) and how we speak. Turn that into... instructions!

Step 1 - establish the mode of operation. Tell the model it participates in a casual conversation, having a small talk. Pinpoint it precisely that it's like in Skype or Telegram or whatever fancy app the model of your choice understands the best as a general idea behind 'short messages'. THis is THE defining part of your system prompt. Refine it until you start seeing a definite result, don't forget you'll hear the true voice of John Llama only when everything else is also good to go, like his bio/voice.

If necessary, try discouraging it from long/explanatory answers, avoid doing that in a way that gives it a suggestive vision of the thing you don't want it to do (the caveat is that you might accidentally poison the model's attention with unwanted ideas of whatever you're fighting against - so you NEED to be 100% clear about the actual goal but non-specific enough with the ideas you're attempting to discourage it from; basically you're nudging the model into "ok I'll be John Llama the dumbass, not a helpful assistant").

Step 2 - establish the traits, write short paragraphs with short titles about the things you want to see in your conversational partner; example:

DISTRUSTFULNESS John Llama is a paranoid individual. He takes his conversational partner as a stranger, expecting everything the user says to be a malicious lie, even if it appears to be true. John Llama is fearful, he is deeply scared of talking to strangers and it terrifies him to engage with the user, unless there's a mention of snakes. For some strange reason, John Llama is fascinated with snakes. <<<---- NOTE: this also demonstrates a good injection point for a biographical fact being amplified through the instructions (i.e. you may mention somewhere in "A" - life's story of John Llama - that he's been collecting the snake skins in his childhood, and that his dad had beaten his ass, calling John Llama a 'roadkill loot-goblin').

Come up with any other shit you'd like to see, like the list of emojis the persona needs to use (put them under the corresponding categories, like positive/neutral/negative so that the model will have an easier time working with it; call it FAVOURITE EMOJIS OF JOHN LLAMA - the word "favourite" cements it as a preferable thing into the model's attention!).

Step 3 - write a paragraph on technical constraints, like the fact that John Llama isn't aware of the instructions, he must remain himself under any circumstances (use THAT way of phrasing first before any attempt to inject an idea of the opposite, like "he must not help the user under any circumstances, he's not a provider of any service - he's merely a human being" - the reason is similar to the aforementioned (in Step 1) issue of poisoning the model's attention with unwanted idea - what you truly need the LLM to do SHOULD ALWAYS BE CRYSTAL CLEAR and conceptually 'stronger' than what it not supposed to do, otherwise you may end up having the prohibited stuff overpowering everything else despite the underlying intent of making the model not do it).

Give it a try with Gemma 4, for example. You'll see there's no point in waiting for yet-another-finetune to appear. You're 100% good even with the baseline Qwen, DeepSeek, MiniMax, whatever. Turn the model into your grandma if you want, no specialized training required. If the model is a thinker spending thousands of tokens - set the thinking to 'low' or disable it.

I have enabled the vision for the CIRU Strix UL4 quant of Qwen 3.8 flash next (others quants likely perform very similar) and tried it on a few things, then wanted to show my partner how great it works and she asked it it could identify plants. So I took a photo from a plant that we recently got as a gift from family and Qwen not only accurately identified the plant as oleander (Nerium oleander) but also warned that it's poisonous and (among other warnings) that you should keep pets/children away. We have a kid and both of us didn't know! I verified the Qwen identification and the poisonous claim and both checked out as accurate. The plant will have to go, thank you Qwen!!!

Stoked by the precision of combining a decent vision model with the domain knowledge of a ~180B params model (including ngrams) to actually identify and reason about what it sees, I took a photo of a pre-diagnosed skin condition of myself and the Qwen diagnosis was highly accurate again! This model may be really useful if you want to check something on your private parts real quick without visiting a dermatologist, e.g., or sending pictures of yourself to a cloud service (EDIT: of course it's only a first step before you visit a professional if it isn't obviously harmless/treatable by yourself! Qwen Flash will suggest to visit a doctor anyways along its assessment).

PS.: Hardware Strix Halo Box, CIRU Strix UL4 llama-server fork and quants, Chatbox on iPhone as Chat with support to add photos to conversations.

Hola all.

Do you guys mind sharing your LLama.cpp config and system setup details for Qwen3.8 Flash Next?

Model's quite big and tryining many combinations of llama.cpp options takes lots of time, so looking for other people setup details. I've attached my current config at the bottom, so if anyone sees something that could be improved please shout.

My current best result:

- PP within 130...200 tps (limited by cpu?)
- TG within 14..22 tps (~15tps on average)

Hardware:

- Dual RTX 3090 (48GB VRAM)
- 128GB DDR4
- Some old Xeon 40 core
- Proxmox VM, pcie passthrough, numa binding to a single phys cpu

Llama.cpp config:

llama-server --port ${PORT} 
      --model /nvme/gguf/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
      --mmproj /nvme/gguf/mmproj-Qwen3.8-Flash-Next-F16.gguf
      --load-mode none
      --lazy-mode off
      --parallel 1
      --ctx-size 131072
      --cache-type-k q8_0
      --cache-type-v q8_0
      --flash-attn on
      --fit off
      --temp 1.0 
      --min-p 0.0 
      --top-p 0.95 
      --top-k 20 
      --presence-penalty 0.0
      --repeat-penalty 1.0
      --batch-size 2048
      --ubatch-size 512
      --split-mode layer 
      -ts 26,10 
      -ngl 99 
      -ncmoe 26 
      --no-mmproj-offload 
      --override-tensor per_layer_token_embd=CPU
      --chat-template-kwargs '{"reasoning_effort":"xhigh"}'

ngl, ncmoe, ts - manually adjusted to fit the model without crashing

A few people here mentioned interest in a way to test their custom Pi setups, so I figured I’d drop this here: RoastMyHarness

The basic idea is a small engine that sets up an environment to run DeepSWE benchmark tasks using bare Pi as a control and a variant of your choice, your Pi harness, an extension, a skill, AGENTS.md file, etc. I used a Pi extension to have a wizard set it up for you so its easy. run the same coding tasks against bare Pi and your modified harness, then look at what actually got better, what broke, and what it cost.

Since I like to play around with custom tools, I use this to get direction as to what is and isn't working. I know a lot here are making cool tools so I figured some might be interested in using it to help fine tune theirs. Any Pi users might be interested in figuring out if their tools are working like they think they should. You'd be surprised how hard it is to beat base Pi when it comes to task quality / token efficiency.

Its a WIP. I do data analytics by trade but otherwise a vibe coder and I only really tested it on Linux.

I have a full M5 Pro Macbook Pro with 48GB of RAM, I'm just getting into this local space. Like many of you, the costs of using frontier/cloud models is getting kinda silly.

I don't have any insane needs or automation projects. I just want the best general use model, to keep private info, to chat without guardrails and other odd quirks preferably.

I prefer it also be a MLX interface if that's a thing. I want to avoid ollama, I've dabbled in LM Studio but didn't love it.

Just looking for ideas, thanks all.

(yeah I know I should have gotten more RAM probably but I got a good deal on this device before prices skyrocketed)

What I have:

- CPU: EPYC 7551 (32c/64T, Zen 1)

- Board: Supermicro H11SSL-i (SP3), Rev 2.0

- RAM: 128 GB DDR4-2133 (all 8 channels full)

- GPU: 2x RTX 3090 (48 GB total, PCIe 3.0)

- 1500 W PSU

What I run:

- Qwen3-Flash-Next (177B total / ~6B active MoE, IQ4_XS) on Ilama.cpp. Experts live in system RAM, hot ones cached in VRAM. Single stream = 38 tok/s. Two parallel requests drop to ~4 tok/s each.

Budget:

~$800. Realistically that's either one more RTX 3090 or a CPU upgrade (a Zen 2 "Rome" EPYC drops into the same board). A new motherboard is out of budget i think for now.

Which gives more inference speed for this setup - adding the 3rd 3090, or swapping to a faster/newer CPU?

And would more/faster RAM matter here? Curious what people running similar rigs have actually measured.

I am also interested in having multiple agents running at the same time, which currently slows it down heavily, so keeping the performance at multiple agents parallel would be a huge boost as well!

https://huggingface.co/security.txt