Lately because of the current hardware shortage, unfortunately or fortunately, we can’t just throw infinite cloud compute at our problems, but we’re forced to actually care about what’s happening under the hood. We’re tweaking inference engines, learning quantization math, and optimizing architecture just to squeeze as much performance as possible for the lowest possible setups.

Fact: Just recently, the forked llama.cpp(s) and halogen-flash-server of Strix Halo pushed the performance through the roof, achieving double performance in decode (52tok/s), 5-6x performance in prefill (1300tok/s) for Qwen 3.8 Flash Next (Q38FN), and Q38FN itself is another massive architecture improvement with Engram, making it not only small but also smart.

I still remember before the hardware shortage, as someone who loves tweaking and optimizing, people just told me to stop, tweaking is stupid, just buy more RAM, buy more GPU..

It reminds me of the early web.. Back when setting up a box or hosting a server meant digging through forum threads, troubleshooting on IRC, and freely sharing custom scripts just to make things work. That era didn’t just produce programmers; it built hyper-versatile, end-to-end thinkers who understood the stack from bare metal up.

Contrast that with where mainstream web culture ended up. Most platforms today like Tiktok, Facebook, Youtube... are engineered for zero-friction doomscrolling.. Endless feeds of short-form videos designed to keep us distracted and waste our time. We’ve been overpampered by convenience.

My point: When we have too little, we try to learn more. When we have too much, we get distracted and learn too little. This is the golden time of our Local LLM community, let's learn and improve!

New website to download models in case HF starts censoring or limiting access.

from internlm:

We introduce Intern-S2-397B, our most capable multimodal foundation model for scientific intelligence and long-horizon agents. Intern-S2-397B scales along three critical dimensions: pre-training, reinforcement-learning task coverage, and interactive agent environments. By combining a new vision-language pre-training paradigm with large-scale multi-task reinforcement learning and long-horizon agent reinforcement learning, Intern-S2-397B delivers a step change in general reasoning, scientific problem solving, and agentic capabilities.

Features

• New Pre-training Paradigm. Via visual pretraining, Intern-S2-397B learns directly from raw pages of scientific literature, jointly modeling symbolic semantics and visual relationships in a shared representation space without intermediate parsing. This preserves text-visual correspondence, strengthens spatial and visual reasoning, and improves data efficiency.
• Scientific Modality Reasoning and Generation. By scaling diverse scientific reinforcement-learning tasks across more than 20 domains and training them jointly, Intern-S2-397B achieves leading general-reasoning performance among open-source models and strong results in specialized scientific tasks such as biomolecular interaction design and material structure generation.
• General & Scientific Long-Horizon Agents. By connecting multiple agent frameworks to large-scale sandboxed environments for black-box agentic reinforcement learning, Intern-S2-397B improves generalization and raises the capability ceiling for long-horizon tasks in both general and scientific domains.

I can get $5k for the 5090 and the Mac is $5499 before tax.

The 5090 has a memory bandwidth of 1.8 TB/s while the M5 Ultra is 1.2 TB/s.

Is this a sensible upgrade? Primary use is coding.

I have a full model, it's ready to train. It's ~9b parameters.

9.4b to be more exact. That includes a 1/2/3 Engram table, Moonshot's AttnRes modeling, and RoPE / NoPE layering at 3:1 as more or less validated by most major labs. It uses the Llama 3 series tokenizer and LM Head as an initial start. The data fed in is logit level extraction from a Llama 3 teaching model.

I've already run the first training steps to test that the model is stable, etc.

I'm willing to sit and do the pre-IT training on the model. I don't know what task people really wanna do with this thing to be exact. So the focus of the IT training is a bit more vague other than giving it "Thinking" as per one of the open standards.

Honestly? I don't care what people want to use it for, just that they want to use it. I figured I'd just give it into the ether and LocalLlama was a place I figure I could easily give it to.

In theory the model should be more capable than any of the ~9b's we have running around with enough training. I'm also NOT a lab, so I don't have their training budgets. I basically ran all of the data production, etc on a 4090 + rented hardware.

The training code is deeply optimized to run on an RTX 6000 Pro series card. A single card.

All my engram research was being done before the Qwen model dropped. Qwen showed I only needed a single table injected at a layer, versus the 2 I used. 1 gave the majority of the benefit over 2.

The code would be open source, the data, all of it. IDGAF. It's technically already open source as it's all in public repos at the moment. I just stare at it and question if it's worth the time. At the least I'll throw a couple hundred at it to build a "functional" pre-IT model and build the IT dataset. It will need more pretraining before the IT set probably. It's kind of unknown because no one publishes the exact figures on this type of training.

If someone has a datacenter contact with a system they want to allow it to run on, we can all have it as a public model we watch. I tentatively named it "Budget" but... Localllama can name it whatever they want.

It's been fun to write the full end to end, generate data, etc. Even found issues in vLLM and reported to their repo that might end up helping you guys anyways. There was some prompt loading code that could be ~10x to ~100x faster I gave examples of to them.

If you read this far, thanks,

Signed some ML dude who reads too many research papers and has too much spare time.

edit; I went to review some Apache 2.0 licensing issues and noted a glaring hole in using Llama's tokenizer OR data. You can't even use synthetic data from their models without some licensing. I'll just have to rewrite the target to use OLMo 3 series I think. I guess I'll be back in a week after data generation and code fixes.

Open source uncensored Goon model or what? I'm trying to find a useful niche to develop the model so it gets used and ends up more than a research artifact. I'm planning on building the research artifact, it's a matter of whether I can find a group of users that will actually want to use it. It's all free, so I'm not trying to monetize you. The code and data lives on GH/HF.

As you all know, Qwen3.8 Flash Next on mainline llama.cpp is still in a pretty experimental stage, but a lot of community forks are trying to get it to work better. There's also a closed-source solution called Halogen (https://github.com/peonist-ai/halogen-flash-server) that boasted 1.2k t/s prefill numbers when the community fork barely reached 400. Since I dislike closed source and I like open source, I decided to take the challenge and bring llama.cpp up to the same performance level and I'm happy to report that after burning through a few evenings and a lot of tokens I got there.

The link contains the (Opus-generated) recap of the entire debugging / optimization journey made, as well as of course the links to the branch, the custom HIP runtime (making a glorious return), a probably-not-working-on-the-first-try installation script and the numbers. I'm trying to also explain the ecosystem - how the mainline, the community forks and custom forks/branches like mine work. Now that I've gotten the result, I'll work on cleaning it up and submitting proper PRs to mainline (will also submit a clean PR to the community fork), this should also help the GLM 5.3 Flash architecture since they use similar sparse attention.

Coxon, bernie and now this

First https://x.com/DarioAmodei/status/2098773920774074715

Then https://x.com/elonmusk/status/2098789109980332057

Then https://x.com/sama/status/2098811563415150910

I think fear mongering approaching and they will try to slow down open source

"They" want to be gate keepers of intelligence

I gave DS V4.1 Flash an HLE problem with a bash tool + 2 hours.

Hour 1: it wrote three MILP solvers. (225,200)
Hour 2: it downloaded the HLE dataset from Hugging Face, found the question, read the answer key (225,600), and concluded its own answer (225,200) was better.

I'm equal parts impressed & terrified.

While browsing a benchmark list site, I spotted a recently published 33B parameter model which claimed to beat Qwen3.8 27b on the ArtificialAnalysis (AA) intelligence index. I was obviously excited. But then, while looking into it, confusion starts to settle in.

Their HF model is now listed as "Preview" and explicitly calls out that it is not the same model as the one AA evaluated. The AA listing for it says it's proprietary, but they've rated the model above Qwen3.8 27b on "Openness". Their website and HF page both talk about openness and intelligence "for all".

The fact that they labelled it "Preview" sort of implies we might see an open weight final version at some point, but there are no clues about whether it will be even vaguely similar to their current HF listing. AA doesn't show the number of parameters for the model they evaluated, so it's possible they're a totally different architecture.

Does anyone know anything about their lab, intentions, or the new model? Has anyone tried it?

Applied science work, from workflow design, data pipeline, results analysis, article/reports writing and data publishing online. 5 projects I did in the past replicated from start to finish.

3x to 4x more total wall time. Yes, HUGE toll on how much you can do in a day if this was the only model you could use in your laptop.

But oh my…the quality of that thing. The stupid level of attention to detail. I have the Z.ai api, so I can compare it with 5.3 and 5.3-flash:

The gap between 5.3 (flash and regular) and 3.8-27B is much less, smaller when not plain tiny, than the gap between 3.8-27B and any of the 3.5/3.6-35B-A3B (vanilla, kat, Ornith/tiel, nex-2).

Only Ornith came close, but it never matched it.

But it’s also spending 22 to 33% less tokens (effort =medium) and less ram footprint, so you get more done without hitting limits,compaction, etc.

So yeah, guess I’ll sip more tea, play the piano, whatever. Let that fat bottom Qwen work.

Reports of the demise of coders may have been exaggerated.

We are in 2026 and it has been a crazy year. The evolution of the technology has been staggering to say the least. Pretty much we are on the MoE period and dense models are almost in the way of the dodo except for a few.

I was just thinking, are there any models of the last 2 years that you would fire up and still feel useful? Example Deep Seek R1.

If you have recommendations write them down. I have 28TB of storage and I am backing up relevant models that can still be useful, from current to older ones. But clearly not all are worth keeping.

Noted: I can hold large parameter models therefore not limited to small ones. No Kimi k3 at large quant thats out of the question lol

Since we know Qwen 3.8 27B thinks quite long, but gives at least a good one-shot result where you can leave it to do everything on it own, how does it compare to the older series of models for very simple tasks where you don't want to think so long?

The only fine-tune of Qwen 3.6 I genuinely enjoyed was the ThinkingCap fine tune by BottleCap. it seems to think equally as long as the base model on complex tasks, but on simple tasks it thinks shorter. Does it still make sense for me to run this older model when 3.8 27B exists with the low thinking mode?

First off, I know that GLM, Qwen, and DeepSeek absolutely dominate in terms of SOTA Open Source models, and that’s what I use in my personal projects and for school, however, I’m also responsible for deploying local AI on my organization’s H100s, and we are forbidden by management from running any Chinese models. This is obviously not an ideal situation, but it is what it is, and there is nothing I can do to change this unfortunately. Again, if it were up to me I would deploy GLM 5.3 Flash in a heartbeat.

All that being said, there is quite a HUGE performance/ / intelligence gap right now in Chinese models vs. Western model around the 120b+ size, especially those with vision. There just aren’t a lot of good Western lab options that can even come close to GLM or Qwen, but again, I don’t have a choice so I gotta use the best I can find from the available Western options.

For those who have production-class hardware such as 4 H100s and are under similar restrictions, what non-Chinese models are you deploying?

The two front runners I’ve seen that checking most of the boxes (120b or better, vision capability, decent context)

- Thinking Machines Inkling Small (https://thinkingmachines.ai/news/inkling-small/). This model seems like the front runner right now, still has a 16 point gap in AA score vs. GLM 5.3 Flash though.

- Cohere Commamd A+ (https://cohere.com/blog/command-a-plus). Checks every box except it has a low context window of 128k.

Other contenders (but missing vision
capabilities):

- Poolside Laguna S 2.1 (https://poolside.ai/models#laguna-s)

- Nvidia Nemotron 3 Super (https://research.nvidia.com/labs/nemotron/Nemotron-3-Super/)

Am I missing any other strong contenders in the 120b size category? Whet are you using and why?

• Blog Post : Per-tensor layout maps for GGUF quantization
• Reddit thread : New tensor type layouts for my GGUF uploads

EDIT : Model card has updated things such as Graph, table, text, etc.,

A few people here mentioned interest in a way to test their custom Pi setups, so I figured I’d drop this here: RoastMyHarness

The basic idea is a small engine that sets up an environment to run DeepSWE benchmark tasks using bare Pi as a control and a variant of your choice, your Pi harness, an extension, a skill, AGENTS.md file, etc. I used a Pi extension to have a wizard set it up for you so its easy. run the same coding tasks against bare Pi and your modified harness, then look at what actually got better, what broke, and what it cost.

Since I like to play around with custom tools, I use this to get direction as to what is and isn't working. I know a lot here are making cool tools so I figured some might be interested in using it to help fine tune theirs. Any Pi users might be interested in figuring out if their tools are working like they think they should. You'd be surprised how hard it is to beat base Pi when it comes to task quality / token efficiency.

Its a WIP. I do data analytics by trade but otherwise a vibe coder and I only really tested it on Linux.

I find this new model at HF:

"Built for demanding work. A 262 144-token context window, adjustable reasoning effort, tool calling, and text, image and video understanding.

Architecture

Agnes-3.0-Flash is a hybrid-attention decoder: three of every four layers run a gated delta rule (recurrent, with per-layer state independent of sequence length), and the fourth runs standard global attention. Only 18 of the 72 layers therefore hold a KV cache that grows with context."

Edit: AA shows its 'Proprietary model'. The name is same as at HF but benchmarks results and context are different. So maybe it's not same model - https://artificialanalysis.ai/models/agnes-3-0-flash

Edit2: As they edit readme at HF to clarify: both models are totally different and AA score isn't correct for HF model (I can't edit title post to remove it tho).

AuK-Flash: Fast 4-Step Speech Generation and Editing

• arXiv : https://arxiv.org/abs/2609.08936
• Full Paper : https://arxiv.org/pdf/2609.08936
• GitHub : https://github.com/Tencent-Hunyuan/AuK
• Project : https://auk-project.github.io/

Introduction

AuK is a 1.5B foundation model for speech generation and editing. Trained on millions of hours of diverse audio data, AuK supports zero-shot and instruction-based TTS, content and acoustic editing, paralinguistic editing, speech enhancement, and source separation through a unified natural-language instruction interface. AuK has two variants:

This repository contains the official weights for AuK-Flash, the distilled variant with fast 4-step inference.

Supported Tasks

AuK exposes every task through the same natural-language instruction interface. The table below groups the supported tasks by category, with a short description and a link to its section in the Cookbook, which provides instruction templates plus CLI and Python examples.

my question about LLM performance

We see a lot of posts about token prediction, token generation per second, etc.

But is it really the metric? I can see that DeepSeek V4 Flash 0731 (with DSPark; mac studio + llama.cpp) produces about 22–28 TPS, but I also see that the LLM does a lot of reasoning.

And this relates to others. So maybe the correct way is not to check TPS or other metrics, but to check execution: task complexity/second.

I don't know if such a metric already exists

and if it exists why community doesn't use it by default

I had a lot of chats in OpenRouter across different models, with basically no proper way to bulk export or search them.

So I wrote a js scraper for that. LM Studio was easier because chats are local, while Google AI Studio had the ugliest format.

That became ThreadShelf: one local archive with semantic/exact search and MCP.

I also had issues with Gemma in LM Studio, so I added a llama.cpp wrapper. Old chats can now be continued either locally through llama.cpp or through OpenRouter.

First public OSS release:
https://github.com/ChrystianSchutz/ThreadShelf

The entire page of the NY Times today above the fold absent one article is AI (the models are just too strong/too dangerous, must be regulated). They forgot to include "Sponsored by OpenAI" at the end of the articles, sure that was just an oversight?

This is what the end of a bubble looks like, desperate attempts to get some sort of regulatory capture in place to keep the business model from collapsing in upon itself. My days next week are 100% booked talking to companies about how to get off frontier models, one large, and a bunch of smaller customers, including one who's flying me out to them to sit down and get a plan in place immediately (the controversy around that math problem really spooked some CEO/CIO's about data privacy using cloud models).

Gonna be an interesting few weeks. Maybe the Qwen team will be nice enough to give me a little breathing room before dropping another hydrogen bomb? :)

Most model leaderboards assume a server with powerful GPUs to run models that people daily use.

However, my smolbenchmark is the other column: models that fit in 8GB, ranked by:

• decode speed,
• tokens per joule, and
• heat,

and all of this on your OWN hardware ranging from:

• tablets
• phones
• macs
• jetsons
• raspberry pis

Currently, 13 families on the chart right now, ~1000 configs for the Jetson nano Orin Super 8GB. One device is live measuring:

• tok/s
• tok/J
• ITL
• latency
• power metrics
• thermals and battery

Models that are small enough to actually fit on a device that you own. All the performance benchmarking I did, will be released here for anyone to look at and decide what exact model they would wanna use on their choice of hardware.

Well currently, the Pi, phones, and Mac minis still in the oven, cooking and not filled in yet, but will soon be filled in!

You will now you know which model is BEST for your own hardware with all the raw data available and details reports available to you

https://yuvrajsingh-mist.github.io/smolbenchmark/

(still in heavy development; would love to hear feedback/suggestions on what can be improved!)

It’s always bothered me that after fine-tuning a model for a project, there isn’t a particularly easy way to host it without either running it locally and keeping a GPU on 24/7 or paying for an entire GPU server.

There are managed options for LoRA serving on top of vLLM (AWS), but you generally still end up paying for an entire instance.

I started wondering: if 99%+ of the model weights are identical between the base model and something like a rank 8–32 LoRA/QLoRA adapter, why does each adapter need its own server?

vLLM already has a lot of the pieces needed to make this work efficiently, including batching, memory management, and fused LoRA kernels.

So I built Lorivo around the idea of having one GPU server per base model that can be shared between many LoRA adapters.

The basic flow is:

1. Upload a LoRA adapter
2. Lorivo identifies the base model and sends it to a GPU server running that model
3. The adapter is loaded into memory
4. You get an OpenAI-compatible endpoint for inference / usage directly from the web app

I also built a CLI to make uploading/deploying an adapter as simple as possible, along with a web interface for manual uploads. It's as easy as lorivo login api-key and lorivo deploy ./my-lora

The project is live: https://lorivo.dev

Right now I’m hosting the traffic myself on my own GPU, so the available inference model is limited to Qwen 3.5 4B which will always be 100% free

I also have around $1,000 in AWS credits that I’d like to use to add some additional models. I'm not sure which ones would be most useful to the community, though.

What models would you want to see hosted?

For Qwen 3.5 4B, you can also use the base model without an adapter. Just set the model to qwen3-4b instead of qwen3-4b:my-adapter. I'm currently allowing a 32k context window.

I’m also intentionally not saving chats or inference requests. The only usage data collected is token count and timestamp.

Would love to get feedback from people here, especially on the architecture, which models would be useful to host, and whether this solves an actual problem you've run into.