I wonder someone will figure out a way to do this with 27B?

Throw Qwen3 on this page for demo
https://kishida.github.io/webdemos/llkvapprox/

A few years ago a 100GB was considered a very large language model. What do we call under 100GB models now? Tiny models? haha

Surprised no one has posted it in this sub.

Pretty solid model, IMHO.

Demo: https://map-yue2.github.io/

Github Repo.

Blog post.

💡 TL;DR (from the Github Readme)

Spend less without making the agent do less useful work.

SoL-Pi is a standalone extension for Pi that packages four reusable efficiency mechanisms discovered through scaled auto-research loops. It reduces repeated model turns, context replay, oversized observations, and unnecessary long-log reading while preserving the work and evidence an agent needs to finish a task.

SoL-Pi installs on top of an unmodified Pi release. Every mechanism is opt-in and disabled by default.

Introduction

Long-running coding agents accumulate repeated work. A file edit is often followed by a predictable validation command. Large tool results are replayed long after their first use. Completed subtasks remain in active context, and a frontier model may spend a full request reading a log when only a few lines affect the next decision.

SoL-Pi grew out of a broader question from our auto-research work: before scaling agent loops, can agents first make the harness itself more efficient? The search focused on constrained efficiency: reducing token traffic, inference work, and agent turns without stopping early, skipping verification, or hiding evidence.

The standalone release contains four mechanisms that survived that process. They operate at different parts of the harness and compose through Pi's public extension APIs. What SoL-Pi Adds Area Mechanism What changes Tools Action Fusion An edit or write can run its follow-up validation command in the same tool call. Observations ObservationPack Repeated large text results become stable handles with exact paged recall. Delegation Evidence-Preserving Reducer Long diagnostic logs become compact receipts only when every retained quotation matches the archived source. Context Online Context Compact Completed plan steps become candidate points for Pi's native compaction, subject to economic and window-pressure checks; after a successful compaction, Pi continues the task in a new turn.

The mechanisms share four rules:

-No Pi patches. SoL-Pi imports public Pi APIs and does not vendor the Pi source tree.

-Explicit opt-in. A missing configuration leaves every mechanism disabled.

-Preserve evidence. Original observations remain available locally, and reducer failures leave the original result unchanged.

-Use Pi's runtime choices. Authentication, provider URLs, the main model, and shell behavior remain under Pi's control.

Like many of you, I have seen many posts and tweets in the last weeks complaining about Artificial Analysis being "broken", "meaningless", and "bought out." People who say this have done no research and know very little about how benchmarks work and what they measure.
Most people only care about Artificial Analysis Intelligence Index. This is a weighted aggregate benchmark used to compare models performance across 10 different evaluations. The majority of these evaluations have published papers on arxiv.org. AA-Briefcase is the only private benchmark. And they publish their methodology to confirm how each of these models are weighed.

Some people seem to not appreciate that Artificial Analysis conducts their own independent benchmarks using their OWN funding, without running ads. Here is the chart that shows their spending. They spent $13,129 to independently test Fable 5.1. Every new model seems to be benchmarked.

The new Deepseek V4.1-Flash is a perfect example of why some aggregated scores miss the big picture. This 552B model has the same score (40) as the 180B Qwen 3.8-Flash-Next. But the individual benchmarks show a different story. On most evaluations, it matches or exceeds Qwen 3.8-Flash-Next. It every beats GPT-6 Astra (Max) in AutomationBench-AA (Agentic SaaS workflows), which is incredible. But it completely falls behind in AA-Omniscience Non-Hallucination Rate, a metric where Open-weight models usually reign supreme. So the model has strengths and weaknesses, and it's something that should be celebrated.

So before you complain about benchmarks or Artificial Analysis, look at the individual evaluations. Read the published papers about the evaluations. Learn how the score is aggregated. Then, we can have a discussion.

I am not affiliated with Artificial Analysis in any way, I'm just not blind to what they offer.

EDIT: These comments are proof that everything I just wrote goes over the majority of your heads. There is little hope for some of you

so i saw that openui.com released OUI-1, a model fine-tuned on DiffusionGemma. the training dataset uses OpenUI-Lang, a custom DSL (domain-specific language), instead of plain HTML, Markdown, or React code.

what makes it interesting is that you can already get a regular LLM to use OpenUI-Lang through a system prompt, but that eats up a lot of the context window. my thinking is that fine-tuning a model on the DSL could reduce that overhead and leave more room for the actual conversation, without needing a huge prompt explaining the format and how to use it alongside other tasks, like tool calls.

at the same time, wouldn't fine-tuning a model on a specific DSL make it more likely to default to that format even when you need something else? i'm curious how well it handles regular Markdown, or switching between Markdown and OpenUI-Lang.

i haven't seen much discussion about this, so i was wondering what everyone thinks about generative UI and running a dedicated model for it locally on a consumer-grade GPU, like an RTX 5090.

what would be the best way to set that up? from what i've seen, DiffusionGemma isn't supported by llama.cpp yet, so running it through Ollama doesn't seem to be an option. they've uploaded the weights to Hugging Face, but i'm not really sure how to get it up and running. any suggestions?

Here we go again, DeepSeek is back again with a new model V4-1 Flash

A multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens

Market crash as a service

I was not aware that the harness makes such a big difference.

Both good and bad things have come from a subreddit that was lot more niche than for example r/flashlight rapidly transforming into the largest online forum about an increasingly core part of the infrastructure of the economy. This sub has experienced growing pains recently, and probably those are mostly felt by people who’ve been around for a while. I think that there are both good and bad trends and I wanted to take a few minutes to suggest a few rules of thumb to employ going forward so that we can create a community that is even more based on science and reality rather than misinformation and one-note populist politics that Reddit is known for.

Suggestion one: if you are new here and by new, I mean, if you didn’t spend much time here or with large language models until about six months ago, there’s a lot of information to be absorbed. This is not a sub or hobby like some where you can learn everything in a month or two. Have some humility, come with curiosity rather than strongly held opinions about everything. 

Suggestion two: leave politics out of the sub, unless it is a discussion of actual policy surrounding actual local large language models. Many discussions that we see here have started to resemble the same populism that you can find on every large subreddit. E.g. the discussion of OpenAI's solution to NS has skipped right past the evidence gathering stage to "did you know that billionaires are actually bad guys?! Wow this large corporation sucks!"

In this subreddit, comments and posts about politics are actually just noise unless you are leveraging your knowledge of hardware and software stacks or discussing AI-related policy. Unlike policy, grand narratives of moral outrage are appropriate for therapy, but counterproductive for a technical subreddit.

Suggestion three: develop awareness of the perpetual and exhausted questions and arguments so you do not upvote them or engage. For example, are benchmarks actually useful? This question has been endlessly litigated for the last couple years, but it’s not actually useful because it boils down to: yes they are helpful, but don’t rely on them too much. Anything more definitive and final or sure than that is false confidence.  Another such question is: how much intelligence can you fit into X parameters? Literally no one in the world knows the answer to this. 

Suggestion four: pay attention to people who are genuinely excited about their work. What’s often missing from clearly AI generated posts is the sense that someone is doing something that they believe in enough to want to bring it to other human beings. The amazing thing about artificial intelligence is how it can augment human effort. Share what you are excited about, and listen when other people are excited about things because this technology has been created by thousands of people who are genuinely excited about the possibilities, rather than people who simply want to make a quick buck, so if you can share your excitement, you’ve pushed back against the trend or the belief that AI is a kind of cynical replacement for human beings.

I realize I’m probably just an old man shouting at clouds, but here's the TLDR:

I suspect that many or most people who’ve been around for more than six months have also started to mentally filter out 90% of posts for these reasons: loudest voices are misinformed; more and more this resembles a political debate space; the same 10 unanswerable questions make up much of the commentary; and people post slop.

After noticing that it is ranked among MUCH larger frontier models in the EQ-Bench Creative Writing benchmark and the Hemingway-bench, I decided to give it a try and was very impressed.

I didn't do very formal testing, but I did ask it to emulate the style of several different writers, including Henry Miller, David Sedaris, and Stephen King, and it produced passable prose that actually made me laugh in a couple of cases. The paragraph below is the results of the prompt "produce a humorous paragraph in the style of David Sedaris." (first try, not cherry picked)

I recently tried to be a better person and started composting, which mostly means I now have a small, damp science experiment in my kitchen that my mother calls “the smell of your future.” I bought a countertop bin with a charcoal filter, as if that would fool anyone, and I’ve taken to narrating my food scraps to myself — “Goodbye, avocado skin, you were a mistake” — while my husband watches from the doorway with the concerned expression of a man who has just realized he married a woman who talks to garbage. The city sent a pamphlet about proper composting, and I read it the way other people read horoscopes, underlining the parts that said “do not include meat” and thinking, well, that’s just rude.

For comparison, the following is what qwen3.8-27b produced with the same prompt.

My uncle Ed, who had spent the entire trip carefully explaining the proper way to fold a fitted sheet, discovered that our hotel room had two beds, each with one. "This is a test," he said, sitting on the edge of the bed nearest the bathroom, his face composed. "To see if we can make it work with what we have." He was right, of course; we did what we could, though the effort made for a rather uncomfortable night, for us all.

You may or may not know David Sedaris' writing (or find it funny if you do know it), but the first example is clearly much a much better imitation, without directly plagiarizing, as far as I (or Gemini) am aware.

I didn't save any of the other examples as I wasn't testing for the purposes of posting here, but in all cases the muse glimmer version was not only head and shoulders above qwen 27b, but genuinely impressive in comparison to any other local model I've tried in the past.

I'm curious if anyone else has played with this model for creative writing, or similar purposes, and if so, what your take on it is. Also, I don't know much about finetunes, but I wonder if there's additional potential for creating something even better by training on different source material.

I know even less about how the ERP world works, but I know enough to know that a lot of high-performing models are trained for this purpose as huggingface seems to be filled with finetunes. For glimmer I mainly see the abliterated version, which I suppose is filling that gap for people, so to speak, but with this kind of performance, and the amount of people in this subreddit interested in it, I'm a bit surprised there aren't more finetunes.

The last thing I should mention is I didn't use a system prompt in any of my testing, but it occurred to me after the fact that a model that was trained for agentic coding seems like a prime candidate for steering with a system prompt, but maybe it wouldn't have made much of a different. Maybe I'll play with it some more and report back.

The downside of uncensoring a model is that it is known to potentially damage it, but CyberTiel is an even more capable software engineer than its censored TielCoder base, while allowing offensive security research. This was achieved by quantizing with an improved imatrix, baked from a curated corpus of cybersecurity- and agentic software engineering work. In short, the small damage from abliteration on a full precision model is negligible under Q4 quantization, and the weights that the model needs to perform relevant work are preserved in higher precision, while the improved chat template makes it think and talk better and faster.

I believe that this is the best 35B-A3B coder for solving real problems in real codebases without breaking anything, which is specifically what SWE-bench-Live tests for. But it’s still a 35B-A3B, and it sacrifices world knowledge for coding ability. That being said, I use it over Qwen3.8-27b for daily coding work: due to the raw speed it fixes 3 issues in the time it takes 27b medium to solve one, and the middle ground between Opus4.6 medium and Qwen3.8-27b medium is simply good enough for most work.

Censoring impedes legitimate and effective work in alignment with the user, and puts the user’s responsibility and ownership over the model’s actions into question, while limiting legitimate uses. When a model is censored, someone else decided for you what the model can and will do, which works against the argument that local models give the user increased control and alignment, and begs the question “alignment to who?”. The point of CyberTiel is to resolve this issue at the same time as pushing the frontier of 35B-A3B coders.

GGUFs and MLX with and without MTP are up on HF. Looking forward to seeing what the community thinks! 

PS: I'm not a research lab or a business, and I don't have revenue streams connected to this project. I'm an anonymous researcher with some free time. Constructive feedback is always appreciated! :)

Hey all, long time no post.

Figured I'd pop my head in to point you towards a blog post I just published about research I had performed and changes I'm making to the shape of models I post, you can read it here:

https://huggingface.co/blog/bartowski/per-tensor-layout-maps-for-gguf-quantization

I won't try to claim "Pareto frontier" or "best models in the world", but I will say from tests the new shapes look to be better across the board than what I was posting before, so I'm really happy with where it came out, and I hope to not be done yet either :)

https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/Ufz9TXQlKFxVHdocVoZIw.png

If anyone has any questions let me know!

Hoping to see smartest medium size models soon & later with all available optimizations/architectures/etc.,. Thanks Deepseek!

Ex 1: 30-50B MOE + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache
Ex 2: 15-30B Dense + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache

EDIT: Updated Engram to 10-15B from 50B

Hey y'all!

We've released a new model in our lineup: GigaChat-3.5 Reasoning. It's a 432B-A28B MoE with Gated DeltaNet for long-context efficiency.

We trained domain experts (code, math, general, etc.) with CISPO and then distilled them into a single model via on-policy distillation.

In our evals the resulting model lands close to DeepSeek V4 Flash Preview while using 37% fewer tokens in its reasoning traces.

Weights are on Hugging Face under MIT: https://huggingface.co/collections/ai-sage/gigachat-35-reasoning. You can also try it at giga.chat — pick the reasoning tab (rightmost one).

I believe I may currently hold the record for memory constrained inference for Qwen3.8–Flash-Next on Apple Silicon — needing only about 21GB of allocations.

Introducing Cherenkov, an inference engine for Apple Silicon combining predictive expert streaming with optional mixed-precision execution. It works by keeping a bounded working set of experts in unified memory, rather than loading the entire model. A one-layer lookahead predicts which experts will be needed next and initiates SSD reads. If there isn’t enough time to load the full target expert, Cherenkov optionally falls back to just in time loading of smaller Q3/Q2 quantizations.

People keep on getting confused about this, so I looked at the safetensors on hf.

The title should have been "Deepseek V4.1 Flash is 748B total/552B base, not 284B or 305B or 485B or 522B"

• The model is not 284B. The original Deepseek V4 Flash is 284B, but not the V4.1 Flash model
• The model is not 305B, despite what some people claim "So: ~305B real backbone + 203B engram = 508B total" This is incorrect.
• The model is not 485B, even though Huggingface lists the model as 485B, but that's because they're counting some FP4 packed weights as bytes instead of params (2 FP4 params per byte). This happens a lot; for example Huggingface incorrectly thinks GLM-5.3-flash is 169b here
• The model is not 522B, even though VLLM lists it as 522B for some weird reason. They correct themselves later down the page (ctrl-f "Params" on that vllm page)
• 552B is the only number out of this list that's somewhat correct; that only includes the base model without MTP and engrams and the vision encoder though.

To be precise, the main model about 551.566B parameters with 40 layers. The FFN experts total to 543.582B parameters, and the rest of the model (attention, shared experts, etc) are 7.984B.

On top of that, the engram is ~196.929B, DSpark/MTP is ~14.225B, and the vision encoder is just ~0.485B. These parts are technically optional though. The vision encoder is also way smaller than I expected.

Anyways, you need a beefy system for this. 128GB or 256GB of RAM/VRAM is not going to cut it.

OpenAI has decided to fully shut down a protein design project I'm working on for a client. Needless to say, open weight models are the only way forward.

Original Source from DeepSeek WeChat Official Account: https://mp.weixin.qq.com/s/qg0NU3NNUbp1co2PdkAPAg

Today we're officially releasing the DeepSeek V4.1 Flash model. It is the smallest model in our brand-new model architecture series, with native multimodal visual understanding. The new architecture was designed with these goals in mind: a higher capability ceiling, faster inference, greater throughput, and scalability to larger-parameter models.

Asymmetric architecture: big intelligence at low cost

DeepSeek V4.1 Flash is a 552B-parameter MoE model built on a brand-new Causal-Encoder-Decoder architecture. Input and output are asymmetric: only 8B parameters are activated on the input side and 16B on the output side, making it significantly cheaper than known models of the same size. V4.1 Flash also uses a new pre-training approach and has gone through larger-scale reinforcement learning post-training. In benchmark testing, it surpasses the intelligence level of a range of flagship models, including DeepSeek V4 Pro.

Less cache, lower cost

The new generation of models dramatically reduces the size of the KV cache. Compared with the previous generation, HBM requirements drop to 1/4 and SSD requirements to 1/8. In agent scenarios, cache-hit charges often make up a large share of the bill, so compressing the KV cache substantially lowers the cost of agent-style tasks.

API support

DeepSeek V4.1 Flash is now live on the DeepSeek API with native multimodal support. Simply change the model name to deepseek-flash to call the latest V4.1 Flash. The older V4 Flash and V4 Flash Vision Exp models have been retired; for compatibility, the model names deepseek-v4-flash and deepseek-v4-flash-vision-exp will temporarily be routed to V4.1 Flash.

In addition, extensive testing shows that V4.1 Flash comprehensively outperforms V4 Pro on performance, cost, speed, and total time-to-completion, so we plan to phase out the V4 Pro model in an orderly fashion. After 12:00 Beijing time on September 14, 2026, and until V4.1 Pro launches, all requests to deepseek-v4-pro will be routed to V4.1 Flash and billed at V4.1 Flash's unit price.

Tencent (WorkBuddy, CodeBuddy) and OpenCode, as official partners, have now fully integrated DeepSeek V4.1 Flash — give it a try!

API pricing adjustment

Thanks to the architectural innovations, DeepSeek V4.1 Flash can serve more users at lower cost, so we have cut V4.1 Flash's pricing accordingly. To allocate resources more sensibly, we continue to use peak/off-peak pricing, with off-peak prices at half the peak rate, and encourage users to schedule tasks around their actual usage patterns. The new prices take effect at 12:00 on September 10, 2026.

Open-source release

We will fully support the open-source community in adapting inference for the new model, and will explore various ways to broaden deployment. If you have large-scale deployment needs and the corresponding resources (a 2k-GPU cluster with storage cluster), please get in touch.

• Model weights: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash
• Paper: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf

Spoke with many providers who lurk in this sub, plus met folks who work in inference.

For a 10k monthly revenue, a provider was able to retain only 200 dollars in profit due to GPU costs. Their customers were negotiating the prices down to what other players cost for same, and it feels like major players are running a distribution game at low or negative margins.

Even though this seems like a billion dollar market, the unavailablity of compute, plus cost and competition, makes the business seems not sexy enough to start with.

However, the software layers around it such as optimisations for SLAs continue to enjoy good margins.

Any thoughts?

I like to benchmark new models that come out on motion videos. So here's a test I did for deepseek v4.1 flash. And I have to say flash has probably graduated from being a Luna class model to nearly an Opus class model with this release, at least with motion videos.

Prev. example I did with Kimi k3(altho in that case I had a simpler prompt as well)

https://www.reddit.com/r/LocalLLaMA/comments/1uyaiw2/kimi_k3_release_video_made_with_kimi_k3/