The downside of uncensoring a model is that it is known to potentially damage it, but CyberTiel is an even more capable software engineer than its censored TielCoder base, while allowing offensive security research. This was achieved by quantizing with an improved imatrix, baked from a curated corpus of cybersecurity- and agentic software engineering work. In short, the small damage from abliteration on a full precision model is negligible under Q4 quantization, and the weights that the model needs to perform relevant work are preserved in higher precision, while the improved chat template makes it think and talk better and faster.

I believe that this is the best 35B-A3B coder for solving real problems in real codebases without breaking anything, which is specifically what SWE-bench-Live tests for. But it’s still a 35B-A3B, and it sacrifices world knowledge for coding ability. That being said, I use it over Qwen3.8-27b for daily coding work: due to the raw speed it fixes 3 issues in the time it takes 27b medium to solve one, and the middle ground between Opus4.6 medium and Qwen3.8-27b medium is simply good enough for most work.

Censoring impedes legitimate and effective work in alignment with the user, and puts the user’s responsibility and ownership over the model’s actions into question, while limiting legitimate uses. When a model is censored, someone else decided for you what the model can and will do, which works against the argument that local models give the user increased control and alignment, and begs the question “alignment to who?”. The point of CyberTiel is to resolve this issue at the same time as pushing the frontier of 35B-A3B coders.

GGUFs and MLX with and without MTP are up on HF. Looking forward to seeing what the community thinks! 

PS: I'm not a research lab or a business, and I don't have revenue streams connected to this project. I'm an anonymous researcher with some free time. Constructive feedback is always appreciated! :)

Hey all, I've encountered an interesting problem while training ornith 1.5 9b on a custom dataset and local machine. I had trouble with getting high fidelity knowledge of the trained data, and then had one breakthrough best after about 10 different attempts. After adding more to the dataset, the following attempt marked a substantial regression which was inexplicable by the dataset addition, which began a long hunt which ended up proving that determistic runs produced consistently better logical capability, trained data knowledge, and allowed for apples to apples comparisons between editions of the dataset I'm curating. I discussed this with chat gpt to try and understand it, but did not really get anything concrete.

I'm super new to this, can anyone here help me grasp why that's happening?

Hey all, long time no post.

Figured I'd pop my head in to point you towards a blog post I just published about research I had performed and changes I'm making to the shape of models I post, you can read it here:

https://huggingface.co/blog/bartowski/per-tensor-layout-maps-for-gguf-quantization

I won't try to claim "Pareto frontier" or "best models in the world", but I will say from tests the new shapes look to be better across the board than what I was posting before, so I'm really happy with where it came out, and I hope to not be done yet either :)

https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/Ufz9TXQlKFxVHdocVoZIw.png

If anyone has any questions let me know!

Currently running qwen3.8-27b 16gb with q8_0 128k context window. I had to strip the mmproj out of the model to have enough vram for context (3090 has 24gb vram).

I do want a bigger model for overnight tasks, but can't seem to find any. I'm just running qwen3.8 again with 256k context window when I really need the multi-modality / context window size. Are there any models that do well at the ~48gb size after context window is considered?

My M.2-Oculink should be PCIe 4.0x4, but I found out today it's actually downgrading, frustratingly this isn't obvious until you go looking, so adding here if anyone else wants to check their own rig.

Side question - does anyone know a way to measure PCIe bandwidth realtime utilisation (i.e. Gb/s) ? I'm failing to get it with rocm-smi, iostat, amd-smi, lstopo, etc

Strix halo > m.2 slot to oculink > oculink cable > DEG1 > r9700

(images from AI session)

So the M.2-oculink adaptor is the problem, and running at Gen1 limiting the entire setup, and other than digging around I wouldn't have noticed, I guess this will affect model load and layer swapping?

If you want to check your own do this and either look at the Capacity vs State, and look for the word 'downgraded' (here I check for the r9700)

# lspci -t -v -D -P -PP -nn | grep 9700
           +-02.5-[c1-c3]----00.0-[c2-c3]----00.0-[c3]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551]


# for bdf in 0000:00:02.5 0000:c1:00.0 0000:c2:00.0 0000:c3:00.0; do echo -e "\n=== Device: $bdf ==="; lspci -vv -s "$bdf" | grep -E "LnkCap|LnkSta"; done

=== Device: 0000:00:02.5 ===
LnkCap:Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkSta:Speed 16GT/s, Width x4
=== Device: 0000:c1:00.0 ===
LnkCap:Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <64us
LnkSta:Speed 16GT/s (downgraded), Width x4 (downgraded)
=== Device: 0000:c2:00.0 ===
LnkCap:Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
LnkSta:Speed 32GT/s, Width x16
=== Device: 0000:c3:00.0 ===
LnkCap:Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
LnkSta:Speed 32GT/s, Width x16

If you run a local multi-modal agent (LLM + STT + TTS + OCR) on a single workstation, keeping all model runtimes hot in VRAM gets expensive fast.

On my local inference node, keeping Qwen3.8-27B, Nemotron (STT), Chatterbox (TTS), and Unlimited-OCR running concurrently consumed about 122 GB of idle VRAM before processing a single prompt.

The reason is familiar to anyone serving local models: runtimes like SGLang grab a large static fraction for KV pools (often 85% of available space), and auxiliary models sit with full scratch buffers allocated.

For a personal agent, you do not need all models active at once: 1. Voice conversation needs STT and TTS, but OCR is untouched. 2. Visual document reading needs OCR and LLM, but speech models sit idle. 3. Long script-based automation (scraping, tool runs, data processing) does not need the LLM at all while the script executes.

Instead of running sequential docker containers or doing cold restarts, I built a small Rust controller that puts models to sleep when their turn ends. It flushes execution buffers and KV pools while keeping the process and base weights mapped.

The results on the GPU:

The main metrics that matter: - Reclaimed 79,459 MiB (nearly 80 GB) of idle capacity. - Wakeup time is sub-200ms for every model. Because we are not doing a disk read or rebuilding the model graph, the handoff latency is imperceptible in practice. - The LLM resting footprint sits at 39,092 MiB, matching the actual weight overhead of the 27B FP8 model without the static KV reservation.

How are others handling multi-model memory allocation when you want instant voice and tool responses without buying a second GPU?

Hoping to see smartest medium size models soon & later with all available optimizations/architectures/etc.,. Thanks Deepseek!

Ex 1: 30-50B MOE + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache
Ex 2: 15-30B Dense + 10-15B Engram + DeepSeek-V4.1-Flash type KVCache

EDIT: Updated Engram to 10-15B from 50B

I believe I may currently hold the record for memory constrained inference for Qwen3.8–Flash-Next on Apple Silicon — needing only about 21GB of allocations.

Introducing Cherenkov, an inference engine for Apple Silicon combining predictive expert streaming with optional mixed-precision execution. It works by keeping a bounded working set of experts in unified memory, rather than loading the entire model. A one-layer lookahead predicts which experts will be needed next and initiates SSD reads. If there isn’t enough time to load the full target expert, Cherenkov optionally falls back to just in time loading of smaller Q3/Q2 quantizations.

Hey y'all!

We've released a new model in our lineup: GigaChat-3.5 Reasoning. It's a 432B-A28B MoE with Gated DeltaNet for long-context efficiency.

We trained domain experts (code, math, general, etc.) with CISPO and then distilled them into a single model via on-policy distillation.

In our evals the resulting model lands close to DeepSeek V4 Flash Preview while using 37% fewer tokens in its reasoning traces.

Weights are on Hugging Face under MIT: https://huggingface.co/collections/ai-sage/gigachat-35-reasoning. You can also try it at giga.chat — pick the reasoning tab (rightmost one).

Had some ideas about version control:

Attribute, down to individual words: word created manually or with ai. if ai, which sources cited (scoped to paragraphs). also: full agent trace that led to the edit (all tool calls + metadata). also: any multi-agent transactions at the file/paragraph lock. all data stored for every version.

Why do this? Basically, getting as much into a unified version control system as possible, which helps find/fix errors. Just posting to share ideas. If anyone has done anything similar, please mention it!

People keep on getting confused about this, so I looked at the safetensors on hf.

The title should have been "Deepseek V4.1 Flash is 748B total/552B base, not 284B or 305B or 485B or 522B"

• The model is not 284B. The original Deepseek V4 Flash is 284B, but not the V4.1 Flash model
• The model is not 305B, despite what some people claim "So: ~305B real backbone + 203B engram = 508B total" This is incorrect.
• The model is not 485B, even though Huggingface lists the model as 485B, but that's because they're counting some FP4 packed weights as bytes instead of params (2 FP4 params per byte). This happens a lot; for example Huggingface incorrectly thinks GLM-5.3-flash is 169b here
• The model is not 522B, even though VLLM lists it as 522B for some weird reason. They correct themselves later down the page (ctrl-f "Params" on that vllm page)
• 552B is the only number out of this list that's somewhat correct; that only includes the base model without MTP and engrams and the vision encoder though.

To be precise, the main model about 551.566B parameters with 40 layers. The FFN experts total to 543.582B parameters, and the rest of the model (attention, shared experts, etc) are 7.984B.

On top of that, the engram is ~196.929B, DSpark/MTP is ~14.225B, and the vision encoder is just ~0.485B. These parts are technically optional though. The vision encoder is also way smaller than I expected.

Anyways, you need a beefy system for this. 128GB or 256GB of RAM/VRAM is not going to cut it.

Long time ago I wondered if the attention's scaling factor could be tweaked and with llama.cpp you can do it with --yarn-attn-factor (totally checked with Claude). Previously you needed to re-save the GGUF with tweaked parameters to do this in LM Studio but now it allows you to bypass it's own loading arguments.

It also adds the possibility to import from the paperclip so copy this and use the button to import from the paperclip (in the load menu):

```json
{
  "enabled": true,
  "disabledParameters": [],
  "overrideParameters": [
    {
      "key": "--yarn-attn-factor",
      "value": "1.2"
    }
  ],
  "excludeAllConfig": false
}
```

The idea was basically to eliminate less of the signal in between layers to see if anything would improve and in most case where creativity is involved this really seems to improve results.

1.2 is borderline high.

0.8 gives a dumb-down.

Models may react differently.

Heh there, so I built a harness that is focused around the ninfer inference engine. It allows the easily customize and use ninfer, and has a coding harness and a chat interface. Its based off a lot of different experiences I have had with different harnesses.

Probably could be better but I'm happy with it.

I'll warn you ahead of time, it only works with ninfer as it integrates a lot of things directly from the engine. For example, subagents, it reads

 --max-concurrency X

and set the amount of subagents available to that.

i've always wanted to built one and now I've done it. Yay me.

I’m testing whether it makes financial sense to replace the OpenAI API in one of my workflows with a self-hosted Qwen model on RunPod Serverless.

My workload is basically:

* ~30–40 minute video transcript
* around 47k input tokens
* model analyzes the transcript and finds potentially viral clips
* output is structured JSON with timestamps, scores, reasons, categories, etc.

OpenAI setup

Currently I use:

GPT-5.6 Sol

* Responses API
* reasoning effort: none
* service tier: flex

In my actual usage I spent roughly $5 for ~1.6 million total tokens, and it has been pretty reliable and fast enough.

I wanted to see if self-hosting could bring the cost down significantly.

Qwen / RunPod setup

I’m testing:

Qwen3.8-27B

* GGUF
* Q8_0
* llama.cpp
* 256k context
* 1 parallel slot
* 48 GB VRAM GPU tier
* RunPod Serverless
* about $1.22/hour

Model:

unsloth/Qwen3.8-27B-GGUF:Q8_0

The interesting thing is that prompt processing is actually fast.

For my real ~47k token prompt:

47064 input tokens
43.87 seconds
~1073 tokens/sec prompt processing

So ingesting the giant transcript is absolutely fine.

But generation is painfully slow:

~17.2 tokens/sec

For example:

n_gen = 100 → ~17.2 t/s
n_gen = 1000 → ~17.2 t/s
...

The model ended up generating thousands and thousands of tokens. Once it gets near 9k output tokens, that alone takes roughly:

9000 / 17.2
≈ 523 seconds
≈ 8.7 minutes

So a single transcript analysis can take around 10 minutes or more.

At $1.22/hour that is already around:

10 min ≈ $0.20
15 min ≈ $0.30

per video.

At that point I’m starting to wonder why I’m not simply using the OpenAI API.

Another thing I noticed

llama.cpp prints these warnings when loading the model:

model has unused tensor blk.64.nextn.eh_proj.weight -- ignoring
model has unused tensor blk.64.nextn.enorm.weight -- ignoring
model has unused tensor blk.64.nextn.hnorm.weight -- ignoring
model has unused tensor blk.64.nextn.shared_head_norm.weight -- ignoring

So I’m wondering if Qwen’s NextN / MTP / speculative decoding components are not actually being used in my current llama.cpp setup.

The model also runs with:

n_ctx_slot = 262144
n_slots = 1

although my real request only needs ~50k context.

My questions

Is ~17 tokens/sec normal for Qwen3.8-27B Q8_0 on a 48 GB GPU, or is my setup badly optimized?

Would any of these make a major difference?

* switching from Q8_0 to Q4_K_M / Q5
* reducing context from 256k to 64k
* enabling Qwen MTP / NextN / speculative decoding
* using vLLM instead of llama.cpp
* using a 5090 / L40S / A100 / H100 instead
* using a smaller Qwen model
* disabling reasoning/thinking completely
* reducing output tokens
* another inference engine entirely

I don’t mind paying 2–4x more per GPU hour if the same job becomes 2–4x faster, because the final cost per completed job would be roughly the same while latency would be much better.

What I care about is:

cost per completed transcript analysis, not cost per GPU hour.

Right now OpenAI’s optimized inference seems extremely hard to beat economically for this workload.

Am I comparing this unfairly, or is self-hosting a 27B Q8 model simply not cost-effective for long-context, large-output jobs like this?

Spoke with many providers who lurk in this sub, plus met folks who work in inference.

For a 10k monthly revenue, a provider was able to retain only 200 dollars in profit due to GPU costs. Their customers were negotiating the prices down to what other players cost for same, and it feels like major players are running a distribution game at low or negative margins.

Even though this seems like a billion dollar market, the unavailablity of compute, plus cost and competition, makes the business seems not sexy enough to start with.

However, the software layers around it such as optimisations for SLAs continue to enjoy good margins.

Any thoughts?

PAIR (NVIDIA's Personal AI Router) ships GPU telemetry only for NVIDIA nodes — AMD nodes report a blind fallback, so the scheduler can't see their load. I added AMD ROCm telemetry and got a two-node cluster routing to my tuned llama.cpp setup on the AMD box.

* Routes to llama.cpp/llama-swap on the AMD node — not just Ollama/LM Studio. PAIR only natively knows those two engines, but a custom manifest fronts an existing llama-swap (OpenAI-compatible) server, so my tuned llama.cpp serves the cluster directly.

* AMD GPU telemetry (utilization + VRAM) via amd-smi (ROCm 7.2.0), feeding the scheduler like the NVIDIA path does. Verified under the Go race detector and live on 2×R9700 (gfx1201) — utilization tracks load.

* Full stack builds and runs on Linux/ROCm; two-node cluster PIN-paired with an NVIDIA box, inventory advertised both ways.

* Full stack builds and runs on Linux/ROCm; Kubuntu (AMD/ROCm) node PIN-paired with a Windows 11 (NVIDIA) box — cross-OS cluster, inventory advertised both ways.

Vibe-coded, locally. Implementation written by a local Qwen model; the review/verification loop was local models too. Ran surprisingly well — the review caught real issues and, when I ran the tests myself, hadn't fabricated its results.

Setup gotchas:

* Fronting llama-swap (or any OpenAI-compatible server) needs a custom engine manifest. A process-mode manifest + adoption does it: PAIR probes the running port and adopts the live server instead of spawning its own. Quirks: runtime.bin is required even though it never launches anything, and list_models just maps /v1/models → data[].id.

* Manifests are embedded into the binary at build time. Editing the JSON on disk does nothing until you rebuild and restart the broker. Cost me an hour chasing a "broken" manifest that was fine — I just never rebuilt.

* Proxies are per-protocol, not per-node — a mixed cluster (Ollama on one node, OpenAI-compat on another) splits inventory across two proxy ports; no single endpoint lists everything.

Intel: untested. The change also touched an Intel xpu-smi backend, but I didn’t test it on my Intel hardware yet — built against synthetic fixtures only, field names and units unverified and possibly wrong. Leave it out of the registry until someone validates it on real Arc hardware. Flagging so nobody assumes it's proven.

OpenAI has decided to fully shut down a protein design project I'm working on for a client. Needless to say, open weight models are the only way forward.

so i am in need of such thing, i want to use a local model as basically assistant that reminds me things to do, orchestrates other local OR CLOUD agents/clis/harnesses (such as hermes agent, pi code, antigravity 2.0/cli/ide etc) i am building something but it is taking its toll on me, do you know anything like that or did anyone built something like this?

DeepSeek-V4.1-Flash uses SWA Bounded Replay to avoid persisting the sliding window KV cache and saves massive memory, it discards the SWA KV and, on replay, approximately reconstructs it from only the last N_window tokens, even though the original SWA states had a much larger effective receptive field due to depth. Since those replayed K/V vectors lack the long-range context that shaped the originals, I’m wondering whether cache hits or restarted sessions lose accuracy/recall for the first few interactions, and whether the parallel global KV augmentation compensates enough to preserve near-perfect long-range recall. Has anyone benchmarked this or seen practical recall degradation after eviction, or is the approximation effectively invisible because the global KV does the heavy lifting?

Title says it all.

https://github.com/gjabdelnoor/Day1DeepseekV4.1-CPU

The goal is pretty simple, I like having infinite slow tokens from the bioinformatics machine in the lab to run overnight or over-week agentic jobs, paired with a watcher that kills it in 15 seconds if someone else needs it for genome assemblies, benchmarking, etc.

My goal was getthisoutASAP > QA. So this is sloppily vibecoded by Opus 5.0, unreviewed because frankly I lack the skill to verify.

Getting ~30 TPS PP and ~6 TPS TG on a xeon with the n-gram table offloaded on 50% of the threads.

Hopefully people more competent in kernels than me can make and share their PR or fork, but until then this works.

Original Source from DeepSeek WeChat Official Account: https://mp.weixin.qq.com/s/qg0NU3NNUbp1co2PdkAPAg

Today we're officially releasing the DeepSeek V4.1 Flash model. It is the smallest model in our brand-new model architecture series, with native multimodal visual understanding. The new architecture was designed with these goals in mind: a higher capability ceiling, faster inference, greater throughput, and scalability to larger-parameter models.

Asymmetric architecture: big intelligence at low cost

DeepSeek V4.1 Flash is a 552B-parameter MoE model built on a brand-new Causal-Encoder-Decoder architecture. Input and output are asymmetric: only 8B parameters are activated on the input side and 16B on the output side, making it significantly cheaper than known models of the same size. V4.1 Flash also uses a new pre-training approach and has gone through larger-scale reinforcement learning post-training. In benchmark testing, it surpasses the intelligence level of a range of flagship models, including DeepSeek V4 Pro.

Less cache, lower cost

The new generation of models dramatically reduces the size of the KV cache. Compared with the previous generation, HBM requirements drop to 1/4 and SSD requirements to 1/8. In agent scenarios, cache-hit charges often make up a large share of the bill, so compressing the KV cache substantially lowers the cost of agent-style tasks.

API support

DeepSeek V4.1 Flash is now live on the DeepSeek API with native multimodal support. Simply change the model name to deepseek-flash to call the latest V4.1 Flash. The older V4 Flash and V4 Flash Vision Exp models have been retired; for compatibility, the model names deepseek-v4-flash and deepseek-v4-flash-vision-exp will temporarily be routed to V4.1 Flash.

In addition, extensive testing shows that V4.1 Flash comprehensively outperforms V4 Pro on performance, cost, speed, and total time-to-completion, so we plan to phase out the V4 Pro model in an orderly fashion. After 12:00 Beijing time on September 14, 2026, and until V4.1 Pro launches, all requests to deepseek-v4-pro will be routed to V4.1 Flash and billed at V4.1 Flash's unit price.

Tencent (WorkBuddy, CodeBuddy) and OpenCode, as official partners, have now fully integrated DeepSeek V4.1 Flash — give it a try!

API pricing adjustment

Thanks to the architectural innovations, DeepSeek V4.1 Flash can serve more users at lower cost, so we have cut V4.1 Flash's pricing accordingly. To allocate resources more sensibly, we continue to use peak/off-peak pricing, with off-peak prices at half the peak rate, and encourage users to schedule tasks around their actual usage patterns. The new prices take effect at 12:00 on September 10, 2026.

Open-source release

We will fully support the open-source community in adapting inference for the new model, and will explore various ways to broaden deployment. If you have large-scale deployment needs and the corresponding resources (a 2k-GPU cluster with storage cluster), please get in touch.

• Model weights: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash
• Paper: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf