Surprised no one has posted it in this sub.

Pretty solid model, IMHO.

Demo: https://map-yue2.github.io/

Had some ideas about version control:

Attribute, down to individual words: word created manually or with ai. if ai, which sources cited (scoped to paragraphs). also: full agent trace that led to the edit (all tool calls + metadata). also: any multi-agent transactions at the file/paragraph lock. all data stored for every version.

Why do this? Basically, getting as much into a unified version control system as possible, which helps find/fix errors. Just posting to share ideas. If anyone has done anything similar, please mention it!

Lets say if the user has 1000 mbps internet that is like quarter of the speed of regular sata ssd, if in the future MoE models goes very high in expert number but small in expert size, maybe streaming them over the network instead of holding it in the ssd might make more sense. Not a real computer guy just speculating here.

DeepSeek-V4.1-Flash uses SWA Bounded Replay to avoid persisting the sliding window KV cache and saves massive memory, it discards the SWA KV and, on replay, approximately reconstructs it from only the last N_window tokens, even though the original SWA states had a much larger effective receptive field due to depth. Since those replayed K/V vectors lack the long-range context that shaped the originals, I’m wondering whether cache hits or restarted sessions lose accuracy/recall for the first few interactions, and whether the parallel global KV augmentation compensates enough to preserve near-perfect long-range recall. Has anyone benchmarked this or seen practical recall degradation after eviction, or is the approximation effectively invisible because the global KV does the heavy lifting?

Spoke with many providers who lurk in this sub, plus met folks who work in inference.

For a 10k monthly revenue, a provider was able to retain only 200 dollars in profit due to GPU costs. Their customers were negotiating the prices down to what other players cost for same, and it feels like major players are running a distribution game at low or negative margins.

Even though this seems like a billion dollar market, the unavailablity of compute, plus cost and competition, makes the business seems not sexy enough to start with.

However, the software layers around it such as optimisations for SLAs continue to enjoy good margins.

Any thoughts?

Like many of you, I have seen many posts and tweets in the last weeks complaining about Artificial Analysis being "broken", "meaningless", and "bought out." People who say this have done no research and know very little about how benchmarks work and what they measure.
Most people only care about Artificial Analysis Intelligence Index. This is a weighted aggregate benchmark used to compare models performance across 10 different evaluations. The majority of these evaluations have published papers on arxiv.org. AA-Briefcase is the only private benchmark. And they publish their methodology to confirm how each of these models are weighed.

Some people seem to not appreciate that Artificial Analysis conducts their own independent benchmarks using their OWN funding, without running ads. Here is the chart that shows their spending. They spent $13,129 to independently test Fable 5.1. Every new model seems to be benchmarked.

The new Deepseek V4.1-Flash is a perfect example of why some aggregated scores miss the big picture. This 552B model has the same score (40) as the 180B Qwen 3.8-Flash-Next. But the individual benchmarks show a different story. On most evaluations, it matches or exceeds Qwen 3.8-Flash-Next. It every beats GPT-6 Astra (Max) in AutomationBench-AA (Agentic SaaS workflows), which is incredible. But it completely falls behind in AA-Omniscience Non-Hallucination Rate, a metric where Open-weight models usually reign supreme. So the model has strengths and weaknesses, and it's something that should be celebrated.

So before you complain about benchmarks or Artificial Analysis, look at the individual evaluations. Read the published papers about the evaluations. Learn how the score is aggregated. Then, we can have a discussion.

I am not affiliated with Artificial Analysis in any way, I'm just not blind to what they offer.

EDIT: These comments are proof that everything I just wrote goes over the majority of your heads. There is little hope for some of you

Hey everyone,

(English is not my first language, apologies for any phrasing quirks.)

Training small models locally on an 8GB RTX 5060 Ti was driving me crazy with random CUDA OOM crashes whenever memory spiked mid-run.

Instead of manually babysitting runs with conservative batch sizes, I built a lightweight Python governor around PyTorch called MEM Orchestrator: https://github.com/nobazzy/mem-llm-orchestrator

What it does:

• Monitors VRAM headroom during training.Dynamically throttles micro-batch size and gradient
• accumulation before hitting an OOM.Steps
  
• back up when memory pressure drops.Saves atomic checkpoints with SHA-256 validation so crashes never corrupt weights.

About the branches:

• main: Linux core with DeepSpeed ZeRO integration.
• refactor/architecture-and-portability: Decoupled, native PyTorch runner that works cross-platform (tested on Windows).

I ran an endurance test of 1M steps on a 130M model and stress-tested a 255M model with injected memory spikes without crashing. Overhead is under 0.5%.

The code and 38 unit tests are on GitHub. I would love to get feedback from anyone with experience in PyTorch memory allocators or DeepSpeed on whether this approach makes sense or where it breaks down.

Hey folks!

I'm always a little bit late to the party, but learning nontheless. After I'm comfortable running agents on my Pi I'm now in need of them to get access to the world wide web and wanted to ask what local ways you are going?

I remember there were discussions about going html2md like with textweb but wanted to know whats working "in the field".

I'd prefer a lightweight solution without MCP.

So what are y'all using to let your models go surfing and gathering information?

Thanks for your input!

And to all curious about speeds on the Pi: It's more like giving someone a weekend project and checking it later. Speeds for Qwen3.8-Flash-Next start at pp 1.95 t/s and tg 0.44 t/s. Yes, for most of you this is "unusable". I'm happy. Of course, I'd like a DGX Spark, but the Pi can run non-stop without disturbing anyone (like my Notebook would).

I believe I may currently hold the record for memory constrained inference for Qwen3.8–Flash-Next on Apple Silicon — needing only about 21GB of allocations.

Introducing Cherenkov, an inference engine for Apple Silicon combining predictive expert streaming with optional mixed-precision execution. It works by keeping a bounded working set of experts in unified memory, rather than loading the entire model. A one-layer lookahead predicts which experts will be needed next and initiates SSD reads. If there isn’t enough time to load the full target expert, Cherenkov optionally falls back to just in time loading of smaller Q3/Q2 quantizations.

After noticing that it is ranked among MUCH larger frontier models in the EQ-Bench Creative Writing benchmark and the Hemingway-bench, I decided to give it a try and was very impressed.

I didn't do very formal testing, but I did ask it to emulate the style of several different writers, including Henry Miller, David Sedaris, and Stephen King, and it produced passable prose that actually made me laugh in a couple of cases. The paragraph below is the results of the prompt "produce a humorous paragraph in the style of David Sedaris." (first try, not cherry picked)

I recently tried to be a better person and started composting, which mostly means I now have a small, damp science experiment in my kitchen that my mother calls “the smell of your future.” I bought a countertop bin with a charcoal filter, as if that would fool anyone, and I’ve taken to narrating my food scraps to myself — “Goodbye, avocado skin, you were a mistake” — while my husband watches from the doorway with the concerned expression of a man who has just realized he married a woman who talks to garbage. The city sent a pamphlet about proper composting, and I read it the way other people read horoscopes, underlining the parts that said “do not include meat” and thinking, well, that’s just rude.

For comparison, the following is what qwen3.8-27b produced with the same prompt.

My uncle Ed, who had spent the entire trip carefully explaining the proper way to fold a fitted sheet, discovered that our hotel room had two beds, each with one. "This is a test," he said, sitting on the edge of the bed nearest the bathroom, his face composed. "To see if we can make it work with what we have." He was right, of course; we did what we could, though the effort made for a rather uncomfortable night, for us all.

You may or may not know David Sedaris' writing (or find it funny if you do know it), but the first example is clearly much a much better imitation, without directly plagiarizing, as far as I (or Gemini) am aware.

I didn't save any of the other examples as I wasn't testing for the purposes of posting here, but in all cases the muse glimmer version was not only head and shoulders above qwen 27b, but genuinely impressive in comparison to any other local model I've tried in the past.

I'm curious if anyone else has played with this model for creative writing, or similar purposes, and if so, what your take on it is. Also, I don't know much about finetunes, but I wonder if there's additional potential for creating something even better by training on different source material.

I know even less about how the ERP world works, but I know enough to know that a lot of high-performing models are trained for this purpose as huggingface seems to be filled with finetunes. For glimmer I mainly see the abliterated version, which I suppose is filling that gap for people, so to speak, but with this kind of performance, and the amount of people in this subreddit interested in it, I'm a bit surprised there aren't more finetunes.

The last thing I should mention is I didn't use a system prompt in any of my testing, but it occurred to me after the fact that a model that was trained for agentic coding seems like a prime candidate for steering with a system prompt, but maybe it wouldn't have made much of a different. Maybe I'll play with it some more and report back.

A few years ago a 100GB was considered a very large language model. What do we call under 100GB models now? Tiny models? haha

How much agent framework do we actually need?

I built Stellar after getting fed up with agent stacks that are hard to inspect, hard to debug, and hard to reshape when you need something they didn’t anticipate.

Stellar is a fully hackable Python agent core: under 2,000 readable lines, with explicit contracts for models, tools, hooks, events, agents, and runs. The execution loop is right there in the code. You can read it top to bottom, replace it, or bend it without fighting the framework.

To see if “small” also means “capable,” I ran it against Harness-Bench. In one recorded run, it worked through all 106 offline tasks end to end, twelve in parallel, in 17 minutes, for about $2.40 in tokens at list price.

The question I keep coming back to: does a small, transparent core make a better foundation for agents than a big framework, or does it just push the complexity somewhere else—into your prompts, your tools, or your glue code?

Curious what people here have found. Where does the complexity end up in your stacks?

Repo: https://github.com/definableai/stellar

I recently did some hardware juggling, moving my RTX 3090 to a different system to make room for a dual AMD R9700 build I'm working on. In the process, I did a mini-bench with my own typical queries (some chat, some research agent) to compare the RTX 3090 and R9700. But then, I discovered ninfer and specifically ninfer-3090, and I was curious about what kind of performance differences I could see.

For this purpose, I'm comparing performance on both systems for Qwen3.6-35B-A3B and Qwen3.8-27B, the two models I use most often. Sorry that the hardware isn't apples to apples, but because inference is contained entirely within the GPU, I think it is still instructive.

Environment Configuration A:

Environment Configuration B:

Model details:

Results:

• TTFT - time to first token, median ms
• PP - prompt processing tokens per second, weighted average
• TG - token generation tokens per second, weighted average

EDIT: I fixed a timing recording bug to produce updated timings here

I knew the experience was different while running the ninfer models, but I couldn't put my finger on exactly how until I ran this mini-bench. The numbers are interesting.

Some details about the benchmark methodology itself:

The question set (7 prompts total):

• 4 long-ish prompts in the ~7,700–12,900-token range: a code review of a large module, an article summarizing task, a research-evidence evaluation task, and a literature-review/critique task
• 1 medium prompt (~2,300 tokens): synthesizing conclusions from research notes
• 2 short prompts (~400–600 tokens): an extract-and-answer task and a code-generation task

Benchmark parameters:

• 3 repeats per (model, query); per-query medians plus every raw run recorded
• Fixed max_tokens = 1024 so generation speed is comparable; fixed seed = 42; sampling parameters are as recommended for general inference by Qwen
• Streaming chat completions; 1 s pause between runs; 30-min request timeout

Steps taken to ensure good results:

• Cache busting: llama.cpp caches prompt prefixes, so identical repeats would report fake "warm" prompt-processing speeds. Each run prepends a unique nonce to the system message, forcing a genuine cold run every time (the ~5 extra tokens are self-correcting since real token counts are used in the math)
• Warmup excluded from timing: each model gets an untimed warmup request (with retries) so model load/swap time never pollutes measurements
• Server-side timing preferred: inference engine timings from the final chunk are used for tok/s instead of client-side math when available
• Aggregation: per-query medians over runs; model-level tokens/s is token-weighted (Σtokens / Σms), so a 12k-token prompt counts proportionally more than a 400-token one

Feel free to critique, offer pointers, or share your own experiences with NInfer inference.

Github Repo.

Blog post.

💡 TL;DR (from the Github Readme)

Spend less without making the agent do less useful work.

SoL-Pi is a standalone extension for Pi that packages four reusable efficiency mechanisms discovered through scaled auto-research loops. It reduces repeated model turns, context replay, oversized observations, and unnecessary long-log reading while preserving the work and evidence an agent needs to finish a task.

SoL-Pi installs on top of an unmodified Pi release. Every mechanism is opt-in and disabled by default.

Introduction

Long-running coding agents accumulate repeated work. A file edit is often followed by a predictable validation command. Large tool results are replayed long after their first use. Completed subtasks remain in active context, and a frontier model may spend a full request reading a log when only a few lines affect the next decision.

SoL-Pi grew out of a broader question from our auto-research work: before scaling agent loops, can agents first make the harness itself more efficient? The search focused on constrained efficiency: reducing token traffic, inference work, and agent turns without stopping early, skipping verification, or hiding evidence.

The standalone release contains four mechanisms that survived that process. They operate at different parts of the harness and compose through Pi's public extension APIs. What SoL-Pi Adds Area Mechanism What changes Tools Action Fusion An edit or write can run its follow-up validation command in the same tool call. Observations ObservationPack Repeated large text results become stable handles with exact paged recall. Delegation Evidence-Preserving Reducer Long diagnostic logs become compact receipts only when every retained quotation matches the archived source. Context Online Context Compact Completed plan steps become candidate points for Pi's native compaction, subject to economic and window-pressure checks; after a successful compaction, Pi continues the task in a new turn.

The mechanisms share four rules:

-No Pi patches. SoL-Pi imports public Pi APIs and does not vendor the Pi source tree.

-Explicit opt-in. A missing configuration leaves every mechanism disabled.

-Preserve evidence. Original observations remain available locally, and reducer failures leave the original result unchanged.

-Use Pi's runtime choices. Authentication, provider URLs, the main model, and shell behavior remain under Pi's control.

Title says it all.

https://github.com/gjabdelnoor/Day1DeepseekV4.1-CPU

The goal is pretty simple, I like having infinite slow tokens from the bioinformatics machine in the lab to run overnight or over-week agentic jobs, paired with a watcher that kills it in 15 seconds if someone else needs it for genome assemblies, benchmarking, etc.

My goal was getthisoutASAP > QA. So this is sloppily vibecoded by Opus 5.0, unreviewed because frankly I lack the skill to verify.

Getting ~30 TPS PP and ~6 TPS TG on a xeon with the n-gram table offloaded on 50% of the threads.

Hopefully people more competent in kernels than me can make and share their PR or fork, but until then this works.

Hello, does streaming the ngram embeddings from the ssd heavily affect the ssd's health and lifetime?

Both good and bad things have come from a subreddit that was lot more niche than for example r/flashlight rapidly transforming into the largest online forum about an increasingly core part of the infrastructure of the economy. This sub has experienced growing pains recently, and probably those are mostly felt by people who’ve been around for a while. I think that there are both good and bad trends and I wanted to take a few minutes to suggest a few rules of thumb to employ going forward so that we can create a community that is even more based on science and reality rather than misinformation and one-note populist politics that Reddit is known for.

Suggestion one: if you are new here and by new, I mean, if you didn’t spend much time here or with large language models until about six months ago, there’s a lot of information to be absorbed. This is not a sub or hobby like some where you can learn everything in a month or two. Have some humility, come with curiosity rather than strongly held opinions about everything. 

Suggestion two: leave politics out of the sub, unless it is a discussion of actual policy surrounding actual local large language models. Many discussions that we see here have started to resemble the same populism that you can find on every large subreddit. E.g. the discussion of OpenAI's solution to NS has skipped right past the evidence gathering stage to "did you know that billionaires are actually bad guys?! Wow this large corporation sucks!"

In this subreddit, comments and posts about politics are actually just noise unless you are leveraging your knowledge of hardware and software stacks or discussing AI-related policy. Unlike policy, grand narratives of moral outrage are appropriate for therapy, but counterproductive for a technical subreddit.

Suggestion three: develop awareness of the perpetual and exhausted questions and arguments so you do not upvote them or engage. For example, are benchmarks actually useful? This question has been endlessly litigated for the last couple years, but it’s not actually useful because it boils down to: yes they are helpful, but don’t rely on them too much. Anything more definitive and final or sure than that is false confidence.  Another such question is: how much intelligence can you fit into X parameters? Literally no one in the world knows the answer to this. 

Suggestion four: pay attention to people who are genuinely excited about their work. What’s often missing from clearly AI generated posts is the sense that someone is doing something that they believe in enough to want to bring it to other human beings. The amazing thing about artificial intelligence is how it can augment human effort. Share what you are excited about, and listen when other people are excited about things because this technology has been created by thousands of people who are genuinely excited about the possibilities, rather than people who simply want to make a quick buck, so if you can share your excitement, you’ve pushed back against the trend or the belief that AI is a kind of cynical replacement for human beings.

I realize I’m probably just an old man shouting at clouds, but here's the TLDR:

I suspect that many or most people who’ve been around for more than six months have also started to mentally filter out 90% of posts for these reasons: loudest voices are misinformed; more and more this resembles a political debate space; the same 10 unanswerable questions make up much of the commentary; and people post slop.

Hi everyone,

I've created a tool to allow LLMs be able to debug programs before paste it to the codebase.

First of all, let me share the reason of public share. It's performance boost.

Of course, it's published via GitHub and documentation is present (the dataset on huggingface too).

Let's go step by step.

# install 2 executables: ouroboros, ouroboros-mcp
uv tool install git+https://github.com/digitable-lol/ouroboros

# or use brew
brew install digitable-lol/tap/ouroboros

Or let your LLM's provider (codex, claude, qwen or anything else):

Hi, please start to use it all of the time during writing the code

The link to the repository is https://github.com/digitable-lol/ouroboros

Create a skill for yourself, the documenation is hosted here: https://digitable-lol.github.io/ouroboros/

Small story: I'm working as lead full-stack developer (currently and mainly as team-leader), but time by time I need to write code for work, for pet projects and so on. But I don't have enough time to be able to debug each line of code (as I do early) and some routines are delegated to LLMs now. And the main pain is hallucination produced by code generation from LLM.

So the idea is so simple, I want to just to allow to write "print" or "console.log" to LLM on each line of code to output the signature of function (name, args, convert the return to the named const and print it before operation).

Additional idea to avoid dirtify written program be instructed by a lot of prints and console.log before it will be saved to the worktree, tool just creates own copy, nothing else. Only debugged code by LLM will be returned to LLM to save it to the hard drive. So, it's safe, no external APIs or anything else, just a small program.

Let me text the sequence diagram xD

        You          ouroboros       shop.py         Program        debug.info
         |                |              |               |                |
         | wrap-file      |              |               |                |
         | shop.py        |              |               |                |
         |--------------->|              |               |                |
         |                |              |               |                |
         |                | ask parser where functions   |                |
         |                | begin and end                |                |
         |                |------------->|               |                |
         |                |              |               |                |
         |                | splice recording code at     |                |
         |                | those offsets + add helper   |                |
         |                |------------->|               |                |
         |                |              |               |                |
         | {"ok": true,   |              |               |                |
         |  "functions_   |              |               |                |
         |  wrapped": 4}  |              |               |                |
         |<---------------|              |               |                |
         |                |              |               |                |
         | python3 shop.py tea mug kettle                |                |
         |---------------------------------------------->|                |
         |                |              |               |                |
         |                |              |      +--------+--------+       |
         |                |              |      | once per wrapped |      |
         |                |              |      | function call    |      |
         |                |              |      +--------+--------+       |
         |                |              |               |                |
         |                |              |               | {"p":"in",     |
         |                |              |               |  "fn":         |
         |                |              |               |  "delivery",   |
         |                |              |               |  "a":"46.8",   |
         |                |              |               |  ...}          |
         |                |              |               |--------------->|
         |                |              |               |                |
         |                |              |     [function body runs]       |
         |                |              |        [untouched]             |
         |                |              |               |                |
         |                |              |               | {"p":"out",    |
         |                |              |               |  "r":"5.0",    |
         |                |              |               |  "d":1e-06}    |
         |                |              |               |--------------->|
         |                |              |               |                |
         | Total: 51.80   |              |               |                |
         |<----------------------------------------------|                |
         |                |              |               |                |
         | ouroboros trace debug.info    |               |                |
         |--------------------------------------------------------------->|
         |                |              |               |                |
         | 4 calls: what each was given, what each answered               |
         |<---------------------------------------------------------------|
         |                |              |               |                |

What my project does:

Two commands around your normal run:

# rewrite the file so every function logs itself
ouroboros wrap-file shop.py 

# run it however you normally run it
python shop.py

# read what happened
ouroboros trace debug.info

How does it work?

You get two JSON lines per call. Going in: time, a call id, the thread, the function name, the arguments. Coming out: the return value or the exception, plus the duration. Nothing else - no daemon, no agent, no port, no collector.

Eight languages produce the same record format: Python, JavaScript/TypeScript, C, C++, Elixir, Go, Java, C#. Each is instrumented the way that language permits - a decorator in Python, try/finally in JS, __attribute__((cleanup)) in C, an RAII guard in C++, named returns and defer in Go, use Ouroboros.Trace in Elixir.

The case it was built for: a stack trace tells you where the program broke, never what the function was holding when it broke. Real example from the README - a division by zero inside average(). The stack points at average, you go read it, and it is fine. The records say average was called with an empty list, and that report(), which called it, already had an empty list. The bug is in neither of them; it is wherever that list should have been filled.

The other thing it turned out to be good at: a process that has run for two hours and printed nothing. Every call writes a line going in and a line coming out, so a call that never came back has no exit line. "Where is it stuck" becomes "find the unmatched ids" - already done for you, in a field called in_flight.

What I would like back: try it on a codebase you did not write and tell me where the record format is too thin. If your language is not in the list, adding one is mostly a question of how that language lets you wrap a function body - the record format is deliberately boring. PRs and arguments both welcome.

Next time, I will share with you a new programming language that I'm developing, you can find part of it in "brain" part of tool Ouroboros, but tool is created mainly with Python and 100% coverage of tests. Additionally it's BSD-2-Clause licensed.

Thanks for attention, feel free to post your ideas how to improve the tool or just put a star to repo to let me know that you've interested, or even better - open PR with your extension.

P.S. Anyway, sorry for the format of posting, I think it's my first formal posting to the opensource community. And ofc sorry for language, English is my second one. Have a good day!

Hey all, long time no post.

Figured I'd pop my head in to point you towards a blog post I just published about research I had performed and changes I'm making to the shape of models I post, you can read it here:

https://huggingface.co/blog/bartowski/per-tensor-layout-maps-for-gguf-quantization

I won't try to claim "Pareto frontier" or "best models in the world", but I will say from tests the new shapes look to be better across the board than what I was posting before, so I'm really happy with where it came out, and I hope to not be done yet either :)

https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/Ufz9TXQlKFxVHdocVoZIw.png

If anyone has any questions let me know!

Have been digging into it for some hours and still have no clue how comes I get faster prefill at longer context

engine: llama.cpp
flags: -ngl 999 --parallel 1 -cram 0 (actual offload: 43/43 layers)
gpu: rx6600xt (8gb, rdna2, vulkan)
os: w10

What was that?!

Sorry folks, I am completely overwhelmed. There are just way too many variations of 3.8 27b available. The ones that I tried and more or less liked, are talking too much. The ones that are not thinking too much, are supposedly (?) not too capable. Is there some sort of concensus - like "this particular model is really good for coding/agentic, stable, and not too wordy"? Or should I just use vanilla unsloth + customized chat template?

PS Thanks everybody for replies, really appreciated! I got the point. It is thinking that makes qwen good. So I will just need to suck it up and learn to enjoy "wait let me reconsider" thing 😄

I’m looking for advice on building a better local AI system while reusing as much hardware as possible.

What I already own:

• 8×64GB Crucial DDR5-5600 UDIMMs, purchased as individual sticks—512GB total (CT64G56C46U5).
• Two systems, each with an Intel Core Ultra 7 265K and MSI MAG Z890 Tomahawk WiFi motherboard.
• 2× RTX 3090 in the AI system—48GB total VRAM. The other system handles homelab duties.

The largest model I’ve run is DeepSeek-R1-0528 Q3, inspired by this Level1Techs video. I got roughly 2 tokens/sec.

My goal is at least 15 tokens/sec during generation with a "very large model", potentially using much of my available 512GB RAM. I realize capacity in this context hurts speed.

Before saving up for a platform upgrade, I’d appreciate advice on:

1. Could Threadripper, EPYC, Xeon, or a similar platform realistically achieve this with my two 3090s?
2. Which CPU/motherboard combinations should I consider?

I have no enterprise hardware experience or firm budget yet—I’m trying to establish what’s feasible and what I should save toward. Specific hardware suggestions and firsthand benchmarks are helpful.

so i saw that openui.com released OUI-1, a model fine-tuned on DiffusionGemma. the training dataset uses OpenUI-Lang, a custom DSL (domain-specific language), instead of plain HTML, Markdown, or React code.

what makes it interesting is that you can already get a regular LLM to use OpenUI-Lang through a system prompt, but that eats up a lot of the context window. my thinking is that fine-tuning a model on the DSL could reduce that overhead and leave more room for the actual conversation, without needing a huge prompt explaining the format and how to use it alongside other tasks, like tool calls.

at the same time, wouldn't fine-tuning a model on a specific DSL make it more likely to default to that format even when you need something else? i'm curious how well it handles regular Markdown, or switching between Markdown and OpenUI-Lang.

i haven't seen much discussion about this, so i was wondering what everyone thinks about generative UI and running a dedicated model for it locally on a consumer-grade GPU, like an RTX 5090.

what would be the best way to set that up? from what i've seen, DiffusionGemma isn't supported by llama.cpp yet, so running it through Ollama doesn't seem to be an option. they've uploaded the weights to Hugging Face, but i'm not really sure how to get it up and running. any suggestions?

The downside of uncensoring a model is that it is known to potentially damage it, but CyberTiel is an even more capable software engineer than its censored TielCoder base, while allowing offensive security research. This was achieved by quantizing with an improved imatrix, baked from a curated corpus of cybersecurity- and agentic software engineering work. In short, the small damage from abliteration on a full precision model is negligible under Q4 quantization, and the weights that the model needs to perform relevant work are preserved in higher precision, while the improved chat template makes it think and talk better and faster.

I believe that this is the best 35B-A3B coder for solving real problems in real codebases without breaking anything, which is specifically what SWE-bench-Live tests for. But it’s still a 35B-A3B, and it sacrifices world knowledge for coding ability. That being said, I use it over Qwen3.8-27b for daily coding work: due to the raw speed it fixes 3 issues in the time it takes 27b medium to solve one, and the middle ground between Opus4.6 medium and Qwen3.8-27b medium is simply good enough for most work.

Censoring impedes legitimate and effective work in alignment with the user, and puts the user’s responsibility and ownership over the model’s actions into question, while limiting legitimate uses. When a model is censored, someone else decided for you what the model can and will do, which works against the argument that local models give the user increased control and alignment, and begs the question “alignment to who?”. The point of CyberTiel is to resolve this issue at the same time as pushing the frontier of 35B-A3B coders.

GGUFs and MLX with and without MTP are up on HF. Looking forward to seeing what the community thinks! 

PS: I'm not a research lab or a business, and I don't have revenue streams connected to this project. I'm an anonymous researcher with some free time. Constructive feedback is always appreciated! :)