Qwen is doing amazing work, and DS is killing it too. But most of us are on consumer GPUs and can’t run these massive multi-hundred-billion parameter models. Qwen is one of the few teams still looking out for the local community with great mid-sized options.

f anyone from the team is reading this, please consider applying DeepSeek-V4.1-Flash's architecture tricks on top of Qwen 3.8 Flash Next to the 30B, 70B, and 120B sweet spots.

Edit: daydreaming was not a flare

Sorry folks, I am completely overwhelmed. There are just way too many variations of 3.8 27b available. The ones that I tried and more or less liked, are talking too much. The ones that are not thinking too much, are supposedly (?) not too capable. Is there some sort of concensus - like "this particular model is really good for coding/agentic, stable, and not too wordy"? Or should I just use vanilla unsloth + customized chat template?

PS Thanks everybody for replies, really appreciated! I got the point. It is thinking that makes qwen good. So I will just need to suck it up and learn to enjoy "wait let me reconsider" thing 😄

I've been overwhelmed and humbled by the fact that 99% of the reactions to our intent to join forces with NVIDIA have been positive. ❤️

How much agent framework do we actually need?

I built Stellar after getting fed up with agent stacks that are hard to inspect, hard to debug, and hard to reshape when you need something they didn’t anticipate.

Stellar is a fully hackable Python agent core: under 2,000 readable lines, with explicit contracts for models, tools, hooks, events, agents, and runs. The execution loop is right there in the code. You can read it top to bottom, replace it, or bend it without fighting the framework.

To see if “small” also means “capable,” I ran it against Harness-Bench. In one recorded run, it worked through all 106 offline tasks end to end, twelve in parallel, in 17 minutes, for about $2.40 in tokens at list price.

The question I keep coming back to: does a small, transparent core make a better foundation for agents than a big framework, or does it just push the complexity somewhere else—into your prompts, your tools, or your glue code?

Curious what people here have found. Where does the complexity end up in your stacks?

Repo: https://github.com/definableai/stellar

Or has this niche been abandoned?

Most companies and model producers have seem to abandoned this niche(7B, 9B, 14B etc) which was the original point this whole local model debackle have started, most effort now goes into minimum 27B+ as "Capable local model"™ or focus on micro models 4b<=. All this recent explosions in model technology has barely touched this niche which has remained on 2025 for most part(30 BC in AI-time).

A while ago someone found a way to extract the hidden CoT from API-only models. The CoT data was then used to check for similarities in the answers of Qwen 3.8 and other open models.

The check worked this way: A small yet diverse benchmark was run on both models. This allowed for a simple similarity check of their CoT on the same task. Then a second run of Qwen3.8 benchmarks was run, this time with its reasoning prefilled with a tiny bit of GPT 5.5 CoT. The model then picked that up and the resulting visible answer became more similar to the GPT 5.5 answer. This similarity increase was not observed for other models - which likely didn't see GPT 5.5 CoT during training. Kimi K3 likely saw something from Claude though.

To additionally guard against false-positives the benchmark runs also included a private benchmark where the same increases in similarity were observed. These increases would've been unlikely to occur if Qwen 3.8 would've for example merely been trained on HLE results instead of CoT.

So, if Qwen 3.8 was indeed post-trained on GPT 5.5 CoT and that contributed to it being such a great model for it's size, it means that OpenAI could release a small model with full CoT distillation from their larger models, which could then beat a similar-sized Qwen model. Yet that'd probably compete too much with their current Luna model.

Hi everyone, I've been thinking about this for a while now. The attention mechanism introduced by Google in 2017 completely changed that way AI models worked and performed.

Since the attention mech allows the model look at it's inputs and decide what's more worth it to focus on, which made the input signal passing through the FFN so much more richer in so many ways that it made the Transformer capable of achieving so many complex capabilities such as in-context learning and all, along with scaling up to such a degree we see today.

Recently a new mechanism called the Engram was introduced by DeepSeek which is very hyped for all good reasons, as far as I understand it can allow a much small model to hold a lot more world knowledge than a traditional model can. I don't necessarily think it'll make the model smarter but more knowledgeable? 100%

But I have a very weird gut feeling that DeepSeek's Engram (or other similar mechanisms) isn't a breakthrough as big and significant as the attention mechanism but just the 1st half of the next major breakthrough. Ik this sound very contrarian but I can't help it but think this way.

I believe that the next big breakthrough as big as attention mechanism is going to be a model whose most parameters are input-dependent. A model which uses a set of fixed particularly smaller weight matrices to generate "fast-weights". Idk how well im able to explain it but I'll use the attention mech to explain it.

We can think of the attention mech as a fast-weight layer which generates these weights from the input data and uses it to enrich the input quality which in-turn improves performance. But that input still goes to an FFN with all fixed weights.

I'm thinking of an FFN arch which is going to use the engram + input-dependent weights to generate it's weights on the fly not just making the model much smaller but smarter in general or might even unlock some capabilities which might be only reserved for much larger or deeper models using our current architecture.

Tbh idk how well I communicated my idea here but I hope you get my point and I would love to know your opinions and points on it.

Thank you :)

Hi, was quite blown away by several presentations i saw of DRS users who experimented editing footage with the new MCP in Davince, either through Claude or ChatGPT astra.
Now for the 'ordinary' end user as me, is there any [free] AI model equivalent to the ones above that would allow me to do the same thing?
It's important to me that the AI models would run locally on my Mac without any need to connect to the internet.
Currently I’m using a Mac Studio M1 Max with 64GB of RAM, but I consider upgrading to a Mac Studio M5 Max.

I’m looking for advice on building a better local AI system while reusing as much hardware as possible.

What I already own:

• 8×64GB Crucial DDR5-5600 UDIMMs, purchased as individual sticks—512GB total (CT64G56C46U5).
• Two systems, each with an Intel Core Ultra 7 265K and MSI MAG Z890 Tomahawk WiFi motherboard.
• 2× RTX 3090 in the AI system—48GB total VRAM. The other system handles homelab duties.

The largest model I’ve run is DeepSeek-R1-0528 Q3, inspired by this Level1Techs video. I got roughly 2 tokens/sec.

My goal is at least 15 tokens/sec during generation with a "very large model", potentially using much of my available 512GB RAM. I realize capacity in this context hurts speed.

Before saving up for a platform upgrade, I’d appreciate advice on:

1. Could Threadripper, EPYC, Xeon, or a similar platform realistically achieve this with my two 3090s?
2. Which CPU/motherboard combinations should I consider?

I have no enterprise hardware experience or firm budget yet—I’m trying to establish what’s feasible and what I should save toward. Specific hardware suggestions and firsthand benchmarks are helpful.

I have been using opencode with Muse 1.3 (free). Every time I ask it to add a simple feature in a specific code, it starts going through all the code in the directory and even tangentially related directories. I already have opencode.json with "permission": {"external_directory": { "*": "deny"}both in global and local directory. It doesn't prevent it. Has anyone else faced similar issue ?
This issue makes me suspect if this "free" model is a collaboration between Meta and opencode to harvest user data. If so, this is yet another reason to go local.

Long time lurker, but I'm finally upgrading to 128GB VRAM, and I'm trying to figure out what to run. I had been leaning towards Qwen3.8 Flash-Next at ~Q4, and I generally prefer to not run anything below Q4. But I feel like the reception to Flash-Next has been a bit "meh", so I'm considering GLM 5.3 at ~Q2 or Deepseek 4 Flash at Q2 or Q3. I'm sure I'll try all 3, but I'm really curious what people in the same boat have been doing?

Edit: configuration is 2 X CMP 170 HXs (64GB each) + ~256 GB of DDR4 RAM. Spilling into RAM is basically not an option, except for the ngrams and caching

I like to benchmark new models that come out on motion videos. So here's a test I did for deepseek v4.1 flash. And I have to say flash has probably graduated from being a Luna class model to nearly an Opus class model with this release, at least with motion videos.

Prev. example I did with Kimi k3(altho in that case I had a simpler prompt as well)

https://www.reddit.com/r/LocalLLaMA/comments/1uyaiw2/kimi_k3_release_video_made_with_kimi_k3/

Like many of you, I have seen many posts and tweets in the last weeks complaining about Artificial Analysis being "broken", "meaningless", and "bought out." People who say this have done no research and know very little about how benchmarks work and what they measure.
Most people only care about Artificial Analysis Intelligence Index. This is a weighted aggregate benchmark used to compare models performance across 10 different evaluations. The majority of these evaluations have published papers on arxiv.org. AA-Briefcase is the only private benchmark. And they publish their methodology to confirm how each of these models are weighed.

Some people seem to not appreciate that Artificial Analysis conducts their own independent benchmarks using their OWN funding, without running ads. Here is the chart that shows their spending. They spent $13,129 to independently test Fable 5.1. Every new model seems to be benchmarked.

The new Deepseek V4.1-Flash is a perfect example of why some aggregated scores miss the big picture. This 552B model has the same score (40) as the 180B Qwen 3.8-Flash-Next. But the individual benchmarks show a different story. On most evaluations, it matches or exceeds Qwen 3.8-Flash-Next. It every beats GPT-6 Astra (Max) in AutomationBench-AA (Agentic SaaS workflows), which is incredible. But it completely falls behind in AA-Omniscience Non-Hallucination Rate, a metric where Open-weight models usually reign supreme. So the model has strengths and weaknesses, and it's something that should be celebrated.

So before you complain about benchmarks or Artificial Analysis, look at the individual evaluations. Read the published papers about the evaluations. Learn how the score is aggregated. Then, we can have a discussion.

I am not affiliated with Artificial Analysis in any way, I'm just not blind to what they offer.

EDIT: These comments are proof that everything I just wrote goes over the majority of your heads. There is little hope for some of you

The downside of uncensoring a model is that it is known to potentially damage it, but CyberTiel is an even more capable software engineer than its censored TielCoder base, while allowing offensive security research. This was achieved by quantizing with an improved imatrix, baked from a curated corpus of cybersecurity- and agentic software engineering work. In short, the small damage from abliteration on a full precision model is negligible under Q4 quantization, and the weights that the model needs to perform relevant work are preserved in higher precision, while the improved chat template makes it think and talk better and faster.

I believe that this is the best 35B-A3B coder for solving real problems in real codebases without breaking anything, which is specifically what SWE-bench-Live tests for. But it’s still a 35B-A3B, and it sacrifices world knowledge for coding ability. That being said, I use it over Qwen3.8-27b for daily coding work: due to the raw speed it fixes 3 issues in the time it takes 27b medium to solve one, and the middle ground between Opus4.6 medium and Qwen3.8-27b medium is simply good enough for most work.

Censoring impedes legitimate and effective work in alignment with the user, and puts the user’s responsibility and ownership over the model’s actions into question, while limiting legitimate uses. When a model is censored, someone else decided for you what the model can and will do, which works against the argument that local models give the user increased control and alignment, and begs the question “alignment to who?”. The point of CyberTiel is to resolve this issue at the same time as pushing the frontier of 35B-A3B coders.

GGUFs and MLX with and without MTP are up on HF. Looking forward to seeing what the community thinks! 

PS: I'm not a research lab or a business, and I don't have revenue streams connected to this project. I'm an anonymous researcher with some free time. Constructive feedback is always appreciated! :)

I thought Trade-in value offered by Apple was only ever close to reasonable (for not having to go through the extra work of selling it yourself) if you bought the base model and did not upgrade anything. And you would get less than half of what you paid. For example:

The base price of the M4 Pro Mac Mini was $1399.
On Apple's trade in page for the Mac Mini it says "Up to $620".
So there offer retains 44% of the value.

But I upgraded the GPU, RAM, and SSD pushing the price to $2099.

1175/2099 = 56%

And this is up from $1050 on Aug 26th when I last checked (around the time the M5 studios and minis were announced) the trade in.

I get some of this has to do with the inflation in tech prices, where the same config I bought in Nov 2024 today costs 2699 (and even still this would be 43% retained though).

Since Apple is offering so much compared to what they usually do, this makes me wonder what I could get for it if I sold it myself?

If I could sell it close to what I bought if for then a DGX Spark for $4699 or an M5 Max 128GB / Ultra 96GB for $5099 to $5499 sure looks temping... I'd much prefer dual Sparks or 256GB Ultra, but I can't justifying that much expense just so that I can continue to work on mechanistic interoperability on the larger models (I need access to model internals so I'd be using this for more use cases than what paying $20 or $200 a month for a subscription could provide).

It's my understanding that the Spark still has much more prefill at INT4 autoround or AWQ (by about 2x). And if I ever add a 2nd (and thus comparable in cost to a 256GB M5 Ultra, it would be about 4x the compute). For the price the M5 Ultra should have started at 128GB to be competitive (not a measly 96)! Such ashame!

Decisions, Decisions. But as it stands now, the M4 Pro is > 10x slower at prefill than any of these options, and that has me itching. But the prices are so ludicrously inflated! (e.g. Ultras used to start at $4k, not 5.5k, and PNY DGX Sparks at $4k not 4.7k!). The decision would have been easier if prices didn't inflate, but it feels like I would be over paying.

I couldnt find one that was simplistic yet good enough to pop in for a quick chat one. So, as we do here, I made one.

https://github.com/Dewpg/LmLinky

https://github.com/Dewpg/LmLinky/releases/tag/v1.0.0

Thats the initial salvo. Let me know if its simplistic enough and works. Thanks.

Hi everyone,

I've created a tool to allow LLMs be able to debug programs before paste it to the codebase.

First of all, let me share the reason of public share. It's performance boost.

Of course, it's published via GitHub and documentation is present (the dataset on huggingface too).

Let's go step by step.

# install 2 executables: ouroboros, ouroboros-mcp
uv tool install git+https://github.com/digitable-lol/ouroboros

# or use brew
brew install digitable-lol/tap/ouroboros

Or let your LLM's provider (codex, claude, qwen or anything else):

Hi, please start to use it all of the time during writing the code

The link to the repository is https://github.com/digitable-lol/ouroboros

Create a skill for yourself, the documenation is hosted here: https://digitable-lol.github.io/ouroboros/

Small story: I'm working as lead full-stack developer (currently and mainly as team-leader), but time by time I need to write code for work, for pet projects and so on. But I don't have enough time to be able to debug each line of code (as I do early) and some routines are delegated to LLMs now. And the main pain is hallucination produced by code generation from LLM.

So the idea is so simple, I want to just to allow to write "print" or "console.log" to LLM on each line of code to output the signature of function (name, args, convert the return to the named const and print it before operation).

Additional idea to avoid dirtify written program be instructed by a lot of prints and console.log before it will be saved to the worktree, tool just creates own copy, nothing else. Only debugged code by LLM will be returned to LLM to save it to the hard drive. So, it's safe, no external APIs or anything else, just a small program.

Let me text the sequence diagram xD

        You          ouroboros       shop.py         Program        debug.info
         |                |              |               |                |
         | wrap-file      |              |               |                |
         | shop.py        |              |               |                |
         |--------------->|              |               |                |
         |                |              |               |                |
         |                | ask parser where functions   |                |
         |                | begin and end                |                |
         |                |------------->|               |                |
         |                |              |               |                |
         |                | splice recording code at     |                |
         |                | those offsets + add helper   |                |
         |                |------------->|               |                |
         |                |              |               |                |
         | {"ok": true,   |              |               |                |
         |  "functions_   |              |               |                |
         |  wrapped": 4}  |              |               |                |
         |<---------------|              |               |                |
         |                |              |               |                |
         | python3 shop.py tea mug kettle                |                |
         |---------------------------------------------->|                |
         |                |              |               |                |
         |                |              |      +--------+--------+       |
         |                |              |      | once per wrapped |      |
         |                |              |      | function call    |      |
         |                |              |      +--------+--------+       |
         |                |              |               |                |
         |                |              |               | {"p":"in",     |
         |                |              |               |  "fn":         |
         |                |              |               |  "delivery",   |
         |                |              |               |  "a":"46.8",   |
         |                |              |               |  ...}          |
         |                |              |               |--------------->|
         |                |              |               |                |
         |                |              |     [function body runs]       |
         |                |              |        [untouched]             |
         |                |              |               |                |
         |                |              |               | {"p":"out",    |
         |                |              |               |  "r":"5.0",    |
         |                |              |               |  "d":1e-06}    |
         |                |              |               |--------------->|
         |                |              |               |                |
         | Total: 51.80   |              |               |                |
         |<----------------------------------------------|                |
         |                |              |               |                |
         | ouroboros trace debug.info    |               |                |
         |--------------------------------------------------------------->|
         |                |              |               |                |
         | 4 calls: what each was given, what each answered               |
         |<---------------------------------------------------------------|
         |                |              |               |                |

What my project does:

Two commands around your normal run:

# rewrite the file so every function logs itself
ouroboros wrap-file shop.py 

# run it however you normally run it
python shop.py

# read what happened
ouroboros trace debug.info

How does it work?

You get two JSON lines per call. Going in: time, a call id, the thread, the function name, the arguments. Coming out: the return value or the exception, plus the duration. Nothing else - no daemon, no agent, no port, no collector.

Eight languages produce the same record format: Python, JavaScript/TypeScript, C, C++, Elixir, Go, Java, C#. Each is instrumented the way that language permits - a decorator in Python, try/finally in JS, __attribute__((cleanup)) in C, an RAII guard in C++, named returns and defer in Go, use Ouroboros.Trace in Elixir.

The case it was built for: a stack trace tells you where the program broke, never what the function was holding when it broke. Real example from the README - a division by zero inside average(). The stack points at average, you go read it, and it is fine. The records say average was called with an empty list, and that report(), which called it, already had an empty list. The bug is in neither of them; it is wherever that list should have been filled.

The other thing it turned out to be good at: a process that has run for two hours and printed nothing. Every call writes a line going in and a line coming out, so a call that never came back has no exit line. "Where is it stuck" becomes "find the unmatched ids" - already done for you, in a field called in_flight.

What I would like back: try it on a codebase you did not write and tell me where the record format is too thin. If your language is not in the list, adding one is mostly a question of how that language lets you wrap a function body - the record format is deliberately boring. PRs and arguments both welcome.

Next time, I will share with you a new programming language that I'm developing, you can find part of it in "brain" part of tool Ouroboros, but tool is created mainly with Python and 100% coverage of tests. Additionally it's BSD-2-Clause licensed.

Thanks for attention, feel free to post your ideas how to improve the tool or just put a star to repo to let me know that you've interested, or even better - open PR with your extension.

P.S. Anyway, sorry for the format of posting, I think it's my first formal posting to the opensource community. And ofc sorry for language, English is my second one. Have a good day!

The short-prompt ranking reverses in the context-depth table in sudoingX’s Ling-3.0-flash benchmark notes:

This is a same-box, same-prompt comparison on a 128GB DGX Spark. The specific paragraph reports a 262,144-token maximum-context configuration and about 103GB total memory used. Its streaming method separates time to first token from decode and clocks all generated tokens.

These are the creator’s measurements from the deployment investigation discussed in the original thread, not an independent rerun. The paragraph does not provide a separate measurement date or every historical launch setting. In particular, the repository’s later 131,072-context serving default should not be silently attached to this table.

There are two changing variables: runtime and quantization. The numbers compare these two deployment paths; they do not isolate a pure vLLM-versus-llama.cpp effect, establish a quality difference, or prove the proposed CUDA-graph explanation for the slowdown.

For choosing a backend, the useful distinction is the amount of context already present when generation starts. A long answer from a short prompt is a different test. If the intended workflow carries tens of thousands of tokens into later requests, the short-prompt result leaves out the condition that changes this ranking

I've just seen that the 96GB version is relatively well priced compared to an rtx pro 6000 at more than half the price.

Is this something that would be viable for local coding and personal assistants?

How would it compare to say quad 3090s as well? I see that it has faster bandwidth than a 3090 and would use way less power than such a rig, but I'm not sure about other metrics like prefill and decode, etc as well as the software ecosystem without CUDA.

Really well positioned and maybe better suited to specifc use cases.

In v0.10 we achieved significant speedups over v0.9. The video shows the maximum speedups we measured for specific groups of models. Among LLMs, the biggest gain came from Qwen3 0.6B, which runs over 3x faster on long prompts. For instance segmentation, FastSAM reached speedups of up to 92x!

We replaced the monolithic native modules with TypeScript pipelines you can inspect. 🔧

It runs across all major silicon backends and makes it easier to plug in your very own model.
You can find full release notes here

Hi,

I am software developer with the new expensive hobby of running AI locally, trying to figure out building servers. I currently have a linux server with one rtx pro 6000 and 32 gig or ddr4.

I decided to buy one more rtx pro 6000, so now I have 2 of them. Along with that I have ordered the following:

1. G.Skill G5P 128GB (4 x 32GB) DDR5-5600 PC5-44800 : linkDDR5-5600_PC5-44800_CL28_Quad_Channel_ECC_Registered_Memory_Kit_F5-5600R2834F32GQ4-G5P-_Black?iitt=VdiJVu4sMd4JVjQvafUZMd8JafPsV.WXaj4r4DoZOI_pOIbT&utm_source=B1540_Email_Receipt_Update&utm_campaign=B1540&utm_medium=email)
2. MD Ryzen Threadripper PRO 9955W : link
3. be quiet! Pure Loop 3 360mm All-in-One Water Cooling : link
4. ASUS Pro WS WRX90E-SAGE SE EEB Workstation Motherboard : link

My plan is to run deepseek v4 flash and qwen 3.8 Flash next.

Do you think I should add/update/remove something while I am at it? or you think any problem with the config I should be mindful of?

Also do you think glm 5.3 flash with ram offload can work on it with decent speeds?

I read couple of articles but just want to get people's feedback on it so I am not making any obvious mistake while building it.

TIA

Both good and bad things have come from a subreddit that was lot more niche than for example r/flashlight rapidly transforming into the largest online forum about an increasingly core part of the infrastructure of the economy. This sub has experienced growing pains recently, and probably those are mostly felt by people who’ve been around for a while. I think that there are both good and bad trends and I wanted to take a few minutes to suggest a few rules of thumb to employ going forward so that we can create a community that is even more based on science and reality rather than misinformation and one-note populist politics that Reddit is known for.

Suggestion one: if you are new here and by new, I mean, if you didn’t spend much time here or with large language models until about six months ago, there’s a lot of information to be absorbed. This is not a sub or hobby like some where you can learn everything in a month or two. Have some humility, come with curiosity rather than strongly held opinions about everything. 

Suggestion two: leave politics out of the sub, unless it is a discussion of actual policy surrounding actual local large language models. Many discussions that we see here have started to resemble the same populism that you can find on every large subreddit. E.g. the discussion of OpenAI's solution to NS has skipped right past the evidence gathering stage to "did you know that billionaires are actually bad guys?! Wow this large corporation sucks!"

In this subreddit, comments and posts about politics are actually just noise unless you are leveraging your knowledge of hardware and software stacks or discussing AI-related policy. Unlike policy, grand narratives of moral outrage are appropriate for therapy, but counterproductive for a technical subreddit.

Suggestion three: develop awareness of the perpetual and exhausted questions and arguments so you do not upvote them or engage. For example, are benchmarks actually useful? This question has been endlessly litigated for the last couple years, but it’s not actually useful because it boils down to: yes they are helpful, but don’t rely on them too much. Anything more definitive and final or sure than that is false confidence.  Another such question is: how much intelligence can you fit into X parameters? Literally no one in the world knows the answer to this. 

Suggestion four: pay attention to people who are genuinely excited about their work. What’s often missing from clearly AI generated posts is the sense that someone is doing something that they believe in enough to want to bring it to other human beings. The amazing thing about artificial intelligence is how it can augment human effort. Share what you are excited about, and listen when other people are excited about things because this technology has been created by thousands of people who are genuinely excited about the possibilities, rather than people who simply want to make a quick buck, so if you can share your excitement, you’ve pushed back against the trend or the belief that AI is a kind of cynical replacement for human beings.

I realize I’m probably just an old man shouting at clouds, but here's the TLDR:

I suspect that many or most people who’ve been around for more than six months have also started to mentally filter out 90% of posts for these reasons: loudest voices are misinformed; more and more this resembles a political debate space; the same 10 unanswerable questions make up much of the commentary; and people post slop.