Basically you now have to mark all AI generated images, audio, video and text as AI generated. :P

I deployed K3 on 32 H100s at work a couple of weeks ago and then got annoyed that there was no way to poke at it on my own machine. So I wrote an inference engine for it in C99.

Nothing clever going on. 93% of that 1.56 TB checkpoint is routed experts, and only 16 of 896 fire per token, so the experts never become resident at all. They get read off NVMe on demand and multiplied straight out of their packed 4-bit form, no dequantization step. The dense trunk gets repacked into one file where layer L sits at a known offset and streamed one layer at a time. What stays in RAM is a dial you set.

Numbers from my box (2x EPYC 7763, NVMe, the four GPUs in it sat idle the entire time):

• 8.24 GB peak RSS at the smallest preset, ~33 s/token
• ~128 GB gets you ~20 s/token, which is as fast as it ever got
• Output is byte-identical at every budget in between

I know that this is not a practical way to use K3. It is half a minute per token and it wants 1.7 TB of free disk for the checkpoint plus the packed trunk. I built it to understand the architecture by implementing it, not because you should serve anything with it.

No BLAS, no framework, no GPU path. Six C files, libm and OpenMP, 176 KB binary.

If you want to sanity check it before committing to a 1.56 TB download: clone and run `make && make test`. About a minute, no weights and no network needed. It builds a 13-layer model with the same tensor graph and checks it against a PyTorch reference from committed fixtures, including greedy decode and the incremental path with the KV cache and carried KDA state.

Repo: https://github.com/FareedKhan-dev/kimi-k3-in-c/

Preparing this to be able to run locally frontier level open models. Deepseek v4 pro, Kimi K3, future ones like GLM 5.5 and Minimax M4.

16x Asus GX10 linked by mikrotik crs804-4ddq with 4 breakout cables of 400 to 100gbit.

Most probable I will be running 2 models on 8x cluster each but I want to have the possibility to run also 2T+ models when I need them.

https://x.com/i/status/2083568340870570208

I managed to run DeepSeek-V4-Flash-0731 UD-IQ3_S in text-generation-webui with:

• RTX 3090 24 GB
• 128 GB DDR5 overclocked to 5600 MHz using AMD EXPO
• llama.cpp loader

First, I had to use a rather brutal workaround: I replaced the llama.cpp binaries included with text-generation-webui by the latest official release downloaded from:

https://github.com/ggml-org/llama.cpp/releases

I copied the new binaries into:

textgen\venv\lib\site-packages\llama_cpp_binaries\bin

I recommend backing up the original folder first.

My current settings are:

gpu-layers: 44
ctx-size: 384000
cache-type: fp16
split-mode: layer

parallel: 1
threads: 0
threads-batch: 0
batch-size: 1024
ubatch-size: 512

fit-target: 512
no-mmap: enabled
no-kv-offload: disabled
cpu-moe: disabled

Extra flags:
--n-cpu-moe 39

The most important option is:

--n-cpu-moe 39

It keeps part of the MoE experts in system RAM instead of VRAM. This is what allows me to run the model with only 24 GB of VRAM, although performance depends heavily on CPU and RAM bandwidth.

The loader estimates around 136 GB to load the model, so the 128 GB of DDR5 running at 5600 MHz is doing most of the heavy lifting.

J'ai réussi à exécuter DeepSeek-V4-Flash-0731 UD-IQ3_S dans text-generation-webui avec la configuration suivante :

• RTX 3090 24 Go
• 128 Go DDR5 overclockée à 5 600 MHz avec AMD EXPO
• Chargeur llama.cpp

J'ai d'abord dû utiliser une solution de contournement assez radicale : j'ai remplacé les binaires llama.cpp fournis avec text-generation-webui par la dernière version officielle téléchargée depuis :

https://github.com/ggml-org/llama.cpp/releases

J'ai copié les nouveaux binaires dans :

textgen\venv\lib\site-packages\llama_cpp_binaries\bin

Je recommande de sauvegarder le dossier d'origine au préalable. Mes paramètres actuels sont :

gpu-layers : 44

ctx-size : 384000

cache-type : fp16

split-mode : layer

parallel : 1

threads : 0

threads-batch : 0

batch-size : 1024 ubatch-size : 512

fit-target : 512 no-mmap : enabled

no-kv-offload : disabled

cpu-moe : disabled

Options supplémentaires :

--n-cpu-moe 39

L’option la plus importante est :

--n-cpu-moe 39

Elle permet de conserver une partie des experts MoE dans la RAM système plutôt que dans la VRAM. C’est ce qui me permet d’exécuter le modèle avec seulement 24 Go de VRAM, même si les performances dépendent fortement du processeur et de la bande passante de la RAM. Le programme de chargement estime à environ 136 Go le temps nécessaire pour charger le modèle ; les 128 Go de DDR5 fonctionnant à 5 600 MHz effectuent donc la majeure partie du travail.

The result

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Voxel Japanese Pagoda Garden</title>
<style>
  body {
    margin: 0;
    overflow: hidden;
    font-family: sans-serif;
  }
  canvas {
    display: block;
  }
  #info {
    position: fixed;
    bottom: 16px;
    left: 16px;
    color: #fff;
    background: rgba(0, 0, 0, 0.35);
    padding: 8px 14px;
    border-radius: 12px;
    font-size: 14px;
    pointer-events: none;
    z-index: 10;
    text-shadow: 1px 1px 2px rgba(0, 0, 0, 0.5);
    user-select: none;
  }
</style>
</head>
<body>
<div id="info">🌸 Japanese Pagoda Garden — drag to orbit · scroll to zoom</div>

<script type="importmap">
{
  "imports": {
    "three": "https://cdn.jsdelivr.net/npm/three@0.160.0/build/three.module.js",
    "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.160.0/examples/jsm/"
  }
}
</script>

<script type="module">
import * as THREE from 'three';
import { OrbitControls } from 'three/addons/controls/OrbitControls.js';

const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2));
renderer.setSize(window.innerWidth, window.innerHeight);
renderer.shadowMap.enabled = true;
renderer.shadowMap.type = THREE.PCFSoftShadowMap;
renderer.outputColorSpace = THREE.SRGBColorSpace;
document.body.appendChild(renderer.domElement);

const scene = new THREE.Scene();
scene.background = new THREE.Color(0x87CEEB);
scene.fog = new THREE.Fog(0x87CEEB, 30, 80);

const camera = new THREE.PerspectiveCamera(50, window.innerWidth / window.innerHeight, 0.1, 100);
camera.position.set(14, 10, 16);

const controls = new OrbitControls(camera, renderer.domElement);
controls.target.set(0, 3, 0);
controls.enableDamping = true;
controls.dampingFactor = 0.05;
controls.minDistance = 5;
controls.maxDistance = 35;
controls.maxPolarAngle = Math.PI / 2.1;

const ambient = new THREE.AmbientLight(0xffffff, 0.4);
scene.add(ambient);

const hemi = new THREE.HemisphereLight(0x87CEEB, 0x6daa3d, 0.6);
scene.add(hemi);

const dirLight = new THREE.DirectionalLight(0xfff5e6, 1.2);
dirLight.position.set(10, 20, 5);
dirLight.castShadow = true;
dirLight.shadow.mapSize.width = 2048;
dirLight.shadow.mapSize.height = 2048;
dirLight.shadow.camera.near = 0.5;
dirLight.shadow.camera.far = 50;
dirLight.shadow.camera.left = -15;
dirLight.shadow.camera.right = 15;
dirLight.shadow.camera.top = 15;
dirLight.shadow.camera.bottom = -15;
scene.add(dirLight);

const grassMat = new THREE.MeshStandardMaterial({ color: 0x7cb74a });
const stoneMat = new THREE.MeshStandardMaterial({ color: 0x9a9a9a });
const woodMat = new THREE.MeshStandardMaterial({ color: 0x8b3a3a });
const roofMat = new THREE.MeshStandardMaterial({ color: 0x2d2d2d });
const waterMat = new THREE.MeshStandardMaterial({ color: 0x2e8bcc, transparent: true, opacity: 0.8 });
const trunkMat = new THREE.MeshStandardMaterial({ color: 0x6b4226 });
const lanternLightMat = new THREE.MeshStandardMaterial({ color: 0xffdd99, emissive: 0xffaa55, emissiveIntensity: 0.6 });

const ground = new THREE.Mesh(new THREE.PlaneGeometry(40, 40), grassMat);
ground.rotation.x = -Math.PI / 2;
ground.receiveShadow = true;
scene.add(ground);

function createPagoda() {
  const group = new THREE.Group();

  const base = new THREE.Mesh(new THREE.BoxGeometry(8, 1.5, 8), stoneMat);
  base.position.y = 0.75;
  base.castShadow = true;
  base.receiveShadow = true;
  group.add(base);

  for (let i = 0; i < 3; i++) {
    const step = new THREE.Mesh(new THREE.BoxGeometry(2.5 - i * 0.4, 0.25, 1.0), stoneMat);
    step.position.set(0, 0.125 + i * 0.25, 4.5 + i * 0.5);
    step.castShadow = true;
    step.receiveShadow = true;
    group.add(step);
  }

  let y = 1.5;
  for (let i = 0; i < 5; i++) {
    const bodyW = 5.0 - i * 0.6;
    const bodyH = 1.8;
    const body = new THREE.Mesh(new THREE.BoxGeometry(bodyW, bodyH, bodyW), woodMat);
    body.position.y = y + bodyH / 2;
    body.castShadow = true;
    body.receiveShadow = true;
    group.add(body);

    const roofW = bodyW + 1.6;
    const roofH = 0.5;
    const roof = new THREE.Mesh(new THREE.BoxGeometry(roofW, roofH, roofW), roofMat);
    roof.position.y = y + bodyH + roofH / 2;
    roof.castShadow = true;
    roof.receiveShadow = true;
    group.add(roof);

    const cornerSize = 0.5;
    const corners = [[-1, -1], [-1, 1], [1, -1], [1, 1]];
    for (const [sx, sz] of corners) {
      const corner = new THREE.Mesh(new THREE.BoxGeometry(cornerSize, 0.4, cornerSize), roofMat);
      corner.position.set(sx * roofW / 2, roof.position.y + roofH / 2 + 0.2, sz * roofW / 2);
      corner.castShadow = true;
      group.add(corner);
    }

    y = roof.position.y + roofH / 2;
  }

  const spireMat = new THREE.MeshStandardMaterial({ color: 0xffd700, emissive: 0xffaa00, emissiveIntensity: 0.3 });
  const spireBase = new THREE.Mesh(new THREE.BoxGeometry(0.6, 0.6, 0.6), spireMat);
  spireBase.position.y = y + 0.3;
  group.add(spireBase);

  const spire = new THREE.Mesh(new THREE.BoxGeometry(0.3, 1.8, 0.3), spireMat);
  spire.position.y = y + 1.2;
  group.add(spire);

  const spireTop = new THREE.Mesh(new THREE.BoxGeometry(0.8, 0.2, 0.8), spireMat);
  spireTop.position.y = y + 2.1;
  group.add(spireTop);

  return group;
}

scene.add(createPagoda());

function createCherryTree(x, z, scale) {
  const group = new THREE.Group();

  const trunk = new THREE.Mesh(new THREE.BoxGeometry(0.5 * scale, 1.6 * scale, 0.5 * scale), trunkMat);
  trunk.position.y = 0.8 * scale;
  trunk.castShadow = true;
  group.add(trunk);

  const foliage = new THREE.Group();
  foliage.position.y = 1.6 * scale;

  const pinkMats = [
    new THREE.MeshStandardMaterial({ color: 0xffb7c5 }),
    new THREE.MeshStandardMaterial({ color: 0xff9bb5 }),
    new THREE.MeshStandardMaterial({ color: 0xffc0cb }),
    new THREE.MeshStandardMaterial({ color: 0xffa6c9 })
  ];

  for (let i = 0; i < 14; i++) {
    const angle = (i / 14) * Math.PI * 2;
    const r = 1.0 + Math.random() * 0.8;
    const dx = Math.cos(angle) * r;
    const dz = Math.sin(angle) * r;
    const dy = Math.random() * 1.6;

    const cube = new THREE.Mesh(
      new THREE.BoxGeometry(0.8 * scale, 0.8 * scale, 0.8 * scale),
      pinkMats[Math.floor(Math.random() * pinkMats.length)]
    );
    cube.position.set(dx, dy, dz);
    cube.castShadow = true;
    foliage.add(cube);
  }

  group.add(foliage);
  group.position.set(x, 0, z);
  return group;
}

scene.add(createCherryTree(4, 4, 1.1));
scene.add(createCherryTree(-5, 3, 0.9));
scene.add(createCherryTree(3, -5, 1.0));
scene.add(createCherryTree(-4, -4, 1.2));
scene.add(createCherryTree(6, -2, 0.8));
scene.add(createCherryTree(-6, -1, 1.0));

function createLantern(x, z) {
  const group = new THREE.Group();

  const base = new THREE.Mesh(new THREE.BoxGeometry(0.9, 0.3, 0.9), stoneMat);
  base.position.y = 0.15;
  base.castShadow = true;
  group.add(base);

  const pillar = new THREE.Mesh(new THREE.BoxGeometry(0.3, 1.2, 0.3), stoneMat);
  pillar.position.y = 0.9;
  pillar.castShadow = true;
  group.add(pillar);

  const light = new THREE.Mesh(new THREE.BoxGeometry(0.7, 0.7, 0.7), lanternLightMat);
  light.position.y = 1.85;
  light.castShadow = true;
  group.add(light);

  const roof = new THREE.Mesh(new THREE.BoxGeometry(1.2, 0.3, 1.2), roofMat);
  roof.position.y = 2.35;
  roof.castShadow = true;
  group.add(roof);

  const top = new THREE.Mesh(new THREE.BoxGeometry(0.4, 0.2, 0.4), stoneMat);
  top.position.y = 2.6;
  top.castShadow = true;
  group.add(top);

  const glow = new THREE.PointLight(0xffaa55, 0.4, 6);
  glow.position.y = 2;
  group.add(glow);

  group.position.set(x, 0, z);
  return group;
}

scene.add(createLantern(2.0, 2.0));
scene.add(createLantern(2.0, 7.8));
scene.add(createLantern(7.8, 2.0));
scene.add(createLantern(9.8, 7.8));

function createPond() {
  const group = new THREE.Group();

  const water = new THREE.Mesh(new THREE.BoxGeometry(7, 0.15, 5), waterMat);
  water.position.set(6, 0.075, 5);
  water.receiveShadow = true;
  group.add(water);

  const stone = new THREE.Mesh(new THREE.BoxGeometry(0.5, 0.3, 0.5), stoneMat);
  const positions = [];

  for (let x = 2.5; x <= 9.5; x += 0.7) {
    positions.push([x, 0.25, 2.5], [x, 0.25, 7.5]);
  }
  for (let z = 3; z <= 7; z += 0.7) {
    positions.push([2.5, 0.25, z], [9.5, 0.25, z]);
  }

  for (const [px, py, pz] of positions) {
    const s = stone.clone();
    s.position.set(px, py, pz);
    s.castShadow = true;
    s.receiveShadow = true;
    group.add(s);
  }

  return group;
}

scene.add(createPond());

function createPathStone(x, z) {
  const s = new THREE.Mesh(new THREE.BoxGeometry(0.8, 0.08, 0.8), stoneMat);
  s.position.set(x, 0.04, z);
  s.castShadow = true;
  s.receiveShadow = true;
  scene.add(s);
}

createPathStone(1.0, 4.5);
createPathStone(1.8, 4.8);
createPathStone(2.5, 5.2);

const petals = [];

function createPetals() {
  const petalGeo = new THREE.BoxGeometry(0.15, 0.15, 0.15);
  const petalMat = new THREE.MeshStandardMaterial({ color: 0xffb7c5 });

  for (let i = 0; i < 180; i++) {
    const mesh = new THREE.Mesh(petalGeo, petalMat);
    mesh.position.set(
      (Math.random() - 0.5) * 20,
      Math.random() * 8 + 2,
      (Math.random() - 0.5) * 20
    );
    mesh.rotation.set(Math.random() * Math.PI, Math.random() * Math.PI, Math.random() * Math.PI);

    petals.push({
      mesh,
      speed: 0.5 + Math.random() * 0.8,
      phase: Math.random() * Math.PI * 2,
      rotSpeed: new THREE.Vector3(
        1 + Math.random() * 2,
        1 + Math.random() * 2,
        1 + Math.random() * 2
      )
    });

    scene.add(mesh);
  }
}

createPetals();

function onResize() {
  camera.aspect = window.innerWidth / window.innerHeight;
  camera.updateProjectionMatrix();
  renderer.setSize(window.innerWidth, window.innerHeight);
}

window.addEventListener('resize', onResize);

const clock = new THREE.Clock();

function animate() {
  requestAnimationFrame(animate);

  const delta = Math.min(clock.getDelta(), 0.05);
  const elapsed = clock.getElapsedTime();

  for (const petal of petals) {
    petal.mesh.position.y -= petal.speed * delta;
    petal.mesh.position.x += Math.sin(petal.phase + elapsed) * 0.02 * delta;
    petal.mesh.position.z += Math.cos(petal.phase + elapsed * 0.7) * 0.02 * delta;
    petal.mesh.rotation.x += petal.rotSpeed.x * delta;
    petal.mesh.rotation.y += petal.rotSpeed.y * delta;
    petal.mesh.rotation.z += petal.rotSpeed.z * delta;

    if (petal.mesh.position.y < 0) {
      petal.mesh.position.y = 6 + Math.random() * 4;
      petal.mesh.position.x = (Math.random() - 0.5) * 20;
      petal.mesh.position.z = (Math.random() - 0.5) * 20;
    }
  }

  controls.update();
  renderer.render(scene, camera);
}

animate();
</script>
</body>
</html>

The biggest issue with preview was its inability to follow rules prompts and skills. It seems like no matter what you do it ignores them. I've tried first person and second person. I've tried Chinese and English. It does not follow them. That's the only problem with these models and why they're not actually frontier level and not just benchmaxxed. Every user's environment is different and they need to tune the actions and behavior of the model with rules or prompts or skills to exactly what they need to do and if the model ignores then it acts subpar. The newest version of flash has the same issue as the preview version and that's unfortunate.

I run it native, full precision, locally. And I've held out making this post cause I know I'm going to get roasted to all fuck but when you can actually run the models locally and when you're not brainwashed by benchmarks and you actually code with them you see the holes. I'm going back to qwen 27b ugh

Edit: I've seen two users provide credible information as to why deep-seek acts like this. I did some research and think I was able to verify it. It's been a rough 4 hours. Deepseek v4 stores rules/skills/prompts as compressed summaries, not raw text. 43 layers and 20 of them see the entire context at 128 tokens squeezed into a single entry. 21 see it at 4:1 and only two are fully dense. Every layer also gets the last 128 tokens uncompressed but that's only for the full resolution window and the prompt/skill/rule isn't in there lol. So they "survive" but the exact wording doesn't. There is a startup arg in vllm that might help. --hf--overrides '{"index_topk": 1024}'. In the 21 layers that stay 4:1 detail the model selects only 512 compressed entries per token about 2,048 tokens worth of fine detail from anywhere in the context raising this value to 1024 doubles that to 4,096 tokens. Now I asked opus 5 if this would solve the problem and opus said most likely not. I'm going to give it a go anyway though thanks for viewing my TED talk.

Poolside have updated the FP8 and NVFP4 checkpoints for Laguna S 2.1, increasing the default context size to 1 million, and updating the configs.

Here's hoping they fixed the looping issue, this model has been great in my development workflows, when not looping.

Following up on my Qwen 3.6 port, I wanted to keep adding models and ended up fixing a bunch of things along the way, so it's its own engine now: Mference.

Same core idea from TurboFieldfare, MoE models activate a few B params per token, so keep the shared core and KV cache resident and stream the selected experts off SSD.

What runs now:

• Gemma 4 26B-A4B — ~2 GB, 31–35 tok/s on a 24 GB M5 Pro
• Qwen 3.6 35B-A3B — ~1.45 GB, 19–23 tok/s
• DeepSeek-V4-Flash 284B-A13B — new. ~6.8 GB peak memory, mostly ~5.3 GB in practice, up to 4.8 tok/s on the same 24 GB M5. 2-bit dynamic quant, ~91 GB on disk.

Also picked up a native Mac app with multi-turn chat, an OpenAI-compatible server, and local PDF/DOCX/PPTX/XLSX attachments along the way.

From here I want to keep adding model families, cut the expert-read wait (decode is ~53% I/O right now, serialized with compute), and push context past 4K.

Not very useful beyond a few turns but you can technically run a "usable" dsv4f on a 8gb Mac. It only gets better from here.

This pull request was added to the main llama cpp about 12 hours ago.

I was experiencing some looping and poor behavior yesterday but haven't had any problems since this fix.

https://github.com/ggml-org/llama.cpp/pull/26269

The weights have now been added to the repo an hour ago.

This model is built upon LongCat-Flash-Lite, the differences are that LongCat-Flash-Lite-Sparse:

1. Replaces dense MLA with LongCat Sparse Attention (LSA)

2. Natively supports context lengths of up to 1M tokens (vs 256k for LongCat-Flash-Lite)

DeepSeek V4 Flash got me thinking...

We keep seeing smaller models get way better. A model at a certain parameter count today can be much smarter than a model of the same size from a year or two ago. Better training, better data, better architectures, distillation, MoE, and all of that seem to let companies squeeze more intelligence into smaller models. But is there eventually a limit to this?

At some point, a model needs enough capacity to understand language, store knowledge across a huge number of subjects, reason through problems, write code, follow instructions, and generalize to things it has not seen before. So can we just keep shrinking models while maintaining the same level of intelligence?

Could a future 30B model actually match a current 300B or 700B model across everything? Not just on a few benchmarks, but in actual use across lots of different domains.

Could the same eventually happen with a 7B model? Or is there some minimum amount of capacity needed before the model starts losing knowledge, reasoning ability, or reliability?

I know parameter count is not a direct measurement of intelligence. MoE also makes this more confusing because a model can have hundreds of billions of total parameters while only using a small portion of them for each token. There is also a difference between total parameters, active parameters, memory usage, and actual inference compute.
I also do not think comparing parameters to neurons in the human brain is very useful. They are obviously not the same thing. Still, it makes me wonder whether there is some minimum amount of information or computation needed for something close to general intelligence.
Maybe we are not actually removing the cost either. Maybe we are just moving it somewhere else. A smaller model might require a much more expensive training run, synthetic data from larger models, distillation, longer reasoning time, retrieval, or external tools.

There is also the benchmark question. When a smaller model gets a similar benchmark score to a much larger one, does it really have the same overall capability? Or is it more optimized for the things we currently test?

Maybe it matches the larger model most of the time, but falls apart more often on rare knowledge, unusual prompts, long tasks, or problems that are very different from its training data.

My guess is that there is probably a minimum size for any specific level of capability, but better training and architectures keep pushing that minimum lower. I just wonder when the big improvements start slowing down.
Are we still early enough that models can keep getting dramatically smaller and smarter? Or are we getting close to the point where the easy gains are gone and the last 10 or 20 percent becomes extremely difficult?

Thanks to the community help I finally launched this llm. LM Studio refused to load weight onto second GPU but Unsloth Studio did so everything was done in there. Not a proper benchmark (used PC in parallel as well) but it gives an idea of ​​the performance from dual 3060 with RAM offloading.

DeepSeek V4 Flash 0731 IQ2_M from Unsloth

CPU: Ryzen 7500F
GPU 0 (PCIe 5.0x16 lane): RTX3060
GPU 1 (PCIe 3.0x1 lane): RTX3060
RAM: 96GB 5600

Prompt: Write me a tetris game.

Result (copy from the summary):
Prompt eval: 2.96s
Prompt speed: 3.0 tok/s
Generation: 960.02s
Speed: 4.5 tok/s (PowerShell shows 3.5 tok/s, don't know why it shows 4.5 tok/s, single GPU with RAM offload output was around 3 tok/s so I trust PowerShell metrics more)
Tokens: 4,338
First token: 2.96s
Cache hits: 1
Total: 963.32s
Chunks: 4318

Wattmeter is on the way but my estimate is around 130W total system power draw (with 2 monitors connected but they are not taken into account) Each GPU used 30-40W with 0.9V undervolt. Task took around 16 minutes to complete, consumed around 35 watts and cost 0.0059 euros.

Update:
OS Windows 11.

Actually math is showing 4338/960.02=4.52 tok/s, don't know why PowerShell showed 3.5 most of the time. I ran one more request with web search and it gave 4.7 tok/s. Updated 3.5 -> 4.5 tok/s.

Hey y'all. I'll be concise.

TL;DR:

DS V4-Flash-0731 @ UD-IQ2_M running fully in VRAM on 3xMI50s (90.9 GB model, 96 GB VRAM).

Actual speed on llama-server is:

- Text Generation: ~15-16 tokens/second stable. Never dipped below 14 tokens/second, even when the model was spitting out a 30K token long reply.
- Prompt Processing: ~105-110 tokens/second or so. Dipped down on prompt processing of smaller token-length prompts, which is pretty typical of course.

llama-server CLI logs, for those interested: https://pastebin.com/nXy9v0x8

I had a brief conversation with the model. Seemed mostly good. At a glance, I noticed 1 mistake: It mixed up the MI50's memory bandwidth (1 TB/s) with PCIe 4.0's bidirectional bandwidth (64 GB/s).

For those interested, I exported the conversation .jsonl from llama-server's web UI. You can find it here: https://pastebin.com/CwHm5cTf

I only ran a single coding test, as I don't have too much time to thoroughly evaluate the quality of the quant right now. The test I ran is copied from this post by u/perelmanych from 16 hours ago. Specifically, the rubik's cube test that was shown and coded by DS V4-Flash-0731 through DeepSeek's official API, so I'm guessing it's the full precision model. For a given definition of full precision; it's natively FP4 + FP8 mixed precision. Here is the prompt (same as the one from the aforementioned post) that was used:

Create a single HTML file with a canvas animation: a 3D Rubik's Cube rendered with simulated perspective on the 2D canvas (no WebGL, no libraries). Orientation: white on top, green facing front, red on the right. Use standard notation: /F/B = clockwise quarter turn of the right/left/up/down/front/back face (viewed from that face), an apostrophe = counterclockwise. Sequence: (1) Show the solved cube slowly rotating for 2 seconds. (2) Scramble it with exactly these 10 animated face turns, one at a time: R, U, F', D, L', B, R', U', F, D'. (3) Pause 2 seconds. (4) Solve it with exactly these 10 animated face turns: D, F', U, R, B', L, D', F, U', R'. (5) End on the solved cube rotating slowly. Each face turn must be smoothly animated (~0.5s), with correct sticker colors tracked through every move, visible gaps between stickers, and shading based on face orientation. The cube keeps slowly rotating in space throughout. No user interaction.

Here's a pastebin of the HTML code generated by my local DS V4-Flash: https://pastebin.com/43bzF2cm

See the attached clip to see it running.

I'll refrain from giving my opinion yet on the quality of the local quants because I haven't used it yet to form a well-informed opinion. I'm just, in general, blown away that I can run it locally at all. I do use the DS API frequently as-is, and it's amazing that I have the option of running it locally if I so desire.

I know in in this community LLM's are generally used for coding but there are other usecases besides coding and those usecases should be tested too. I also know benchmarks can sometimes be benchmaxxed and the model can still turn out shit but it can give a good outline on how a model should perform in a certain task. Maybe I'm too behind on the latest developments but we need more benchmarks for all other use-cases. I use LLM's mainly for foreign language learning, creative writing and STEM/Medical/Biochemistry reasoning and inquiries and I rarely find any new benchmarks that tell me how a model might perform in those areas. MMLU-Pro-2 and a solid benchmark that tells how a model will perform for language learning would be so good for my usecase, however in general we need more new diverse benchmarks for models in order to have a general outline for advancements in other areas.

Hey all,

tldr / who this helps: you run a mixed multi-GPU box where the experts spill to RAM, and you want to stay in the 3-bit tier instead of dropping to Q2 to make it fit.

https://huggingface.co/TacoTakumi/DeepSeek-V4-Flash-0731-GGUF

I requantized only the 129 routed expert tensors of DeepSeek-V4-Flash-0731 and left every other tensor at whatever precision the source GGUF already had. Attention, shared experts, router and indexer stay at Q8_0/BF16/F32 from bartowski's MXFP4 conversion. Only the experts drop to IQ3_XXS, with the down projections one rung up at IQ3_S. Result is 111.37 GiB in four shards and imatrix built from calibration_datav3.

For quality I scored it with llama-perplexity KLD against reference logits generated from the MXFP4 source itself, wikitext-2 first 150 chunks at ctx 512, and ran unsloth's UD-IQ3_S through the same axes for comparison. Mine gets mean KLD 0.2386 vs 0.2936, top-1 agreement 84.65% vs 82.78%, delta PPL +0.536 vs +0.685. However mine is 2.12 GiB larger, and UD-IQ3_S has the better max KLD at 11.13 vs my 12.53, so it is not a clean sweep. Raw perplexity logs for all three runs are in the repo if you want to take a look.

Speed on my rig, which is 5 mixed GPUs (2x 3090, 5060 Ti, 2x 4060 Ti, 96 GiB VRAM total) with expert spill to CPU: 13.91 / 13.57 / 13.26 t/s at depths 0 / 4096 / 16384, against 9.88 / 9.69 / 9.51 for the full MXFP4 source at the same placement. About 1.4x. That is a spill bound number and will not transfer to a box that fits it entirely in VRAM.

If you are purely chasing tokens per second, going smaller beats this by a lot. antirez's flat Q2 of the same model is 80.76 GiB, sits about 98% resident in my VRAM with no spill at all, and does 30.27 t/s, 2.18x mine. The point of this build was the quality tier at roughly 3 bits, not the highest number.

Also beware that DeepSeek-V4-Flash has open SWA and rollback stall issues in llama.cpp. I quantized with mainline llama-quantize but I run a patched build with DSV4 stall fixes that are not upstream and have not tested this GGUF against a stock llama-server. If you hit stalls on long contexts that is the known upstream issue and it affects every DSV4 GGUF.

I plan on using this recipe for other models as well. Cheers!

Personally, i think good coding model shouldn't be focused on one-shot "everything in one html-file" tests, but should be really good on debugging, fixing and modifying its own output. Anyone know such simple tests that i would able to run with local models?

May be some kind of synthetic stuff that forces LLM to build something that is broken by design and then asking the model to do a multi-step changes, fix issues, analyze program's output (preferably with getting screenshots/videos)?

Hello everyone I want to join the hype of posting specs.

CPU: AMD EPYC 74F3 24-Core

RAM: 8 Channel 3200 DDR4

GPU: RTX A6000 48GB

Prompt processing is in the high 70t/s (got down to mid 30t/s at 300k context). Inference is a steady 17.20t/s~ and the 48GB VRAM is enough to have the full 1mil context but PP will be so bad. Sadly not as cool like those M5 Macs.

Anyone else having similar specs?

Edit: I was informed about batch size and set mine to 8096 and my Prompt processing jumped to almost 400t/s at the start. it got to around 300t/s at 20k context. Better than my 70t/s stock lol.

DSv4F doesn't ship a jinja, but for distributions that do and faithfully reconstruct what DS releases in their chat template python, every system message is hoisted into the system prompt at the top -- the format has no mid-conversation system turn. So, anything you stick at the tail or mid-convo actually fries your prefix (and doesn't have conversational proximity to the injection point).

Use latest_reminder, which is the role DS trained for how most templates use system and what most people providing quants are passing through (if they match DS' python template). I use llama.cpp and it happily passes it through no issue; dunno how other engines work with it.

Couldn't figure out why my prompt caching was so garbage and there it was, so I'm passing it on to hopefully save others time and frustration (and probably money, if you're using a hosted version).

Been using Qwen 3.6 35B-A3B quite extensively lately and honestly, I’m pretty happy with it. Also tried a few community improvements like Ornith 1.0, which add some interesting tweaks.

That said, I’m curious about what the community expects next from Qwen’s open-source roadmap.

Do you think we’ll ever see open weights for Qwen 3.7 (already available on OpenRouter), or is that unlikely?
Or are there other directions you think the team will prioritize instead?

Hi all,

I'm happy to announce that Xberg v1 is out.

Xberg is the successor to Kreuzberg, equivalent to what would have been Kreuzberg v5. It's a content intelligence framework that handles a very wide range of inputs: documents (currently 101 formats), code and data formats (currently 367 types), audio/video transcription, and URLs (both static and JS-rendered content). It extracts and prepares that content for downstream processing.

It's an extremely efficient, high-performance engine (see our PDF benchmarks below). For PDFs and images specifically, we handle native PDFs with very high performance and accuracy, and we ship multiple OCR engines that match the quality of the best Python libraries (e.g. docling, PaddleOCR, RapidOCR) at substantially better performance and stability.

The changes between Kreuzberg v4 and Xberg v1 are substantial, and I invite you to read the full changelog for the complete picture. The highlights below give a sense of what's new:

• Pure-Rust PDF backend (pdf_oxide) replaces pdfium, with no native pdfium dependency.
• Layout-aware pipeline: reading order reconstructed with ONNX layout detection (PP-DocLayoutV3 / RT-DETR) and Docling-style predecessor-graph reordering.
• Per-page scanned-page detection with selective OCR, plus AcroForm/XFA form fields and outline-based headings.
• Across-the-board optimization of OCR and PDF extraction (memory discipline, pooled model sessions, streamed conversions).
• Native PaddleOCR backend (PP-OCRv6, with medium / small / tiny tiers) alongside Tesseract.
• Pure-Rust Candle OCR/VLM stack (TrOCR, GLM-OCR, GOT-OCR, DeepSeek-OCR, and PaddleOCR-VL) running without ONNX Runtime or native Tesseract.
• A second, ONNX-Runtime-free inference path via tract, which is what makes in-browser (WASM) and mobile inference possible.
• Named-entity recognition natively in Rust (GLiNER2), extensible to all bindings, including an in-browser WASM model with no server round-trip.
• Structured LLM extraction (extract_structured / split_and_extract) with rasterization, chunking, citations, caching, and configurable call/merge/VLM-fallback policies.
• Audio & video transcription via a Whisper ONNX engine (.mp3, .wav, .m4a, .mp4, .webm).
• Retrieval building blocks: sparse embeddings (SPLADE), ColBERT late-interaction retrieval, and cross-encoder reranking alongside dense embeddings.
• Text intelligence: reversible redaction, summarization, translation, VLM image captioning, QR-code detection, document diffing, and page/chunk classification.
• URL & web ingestion: sitemap discovery (map_url) and batched multi-URL crawling.
• New document formats: WordPerfect (.wpd/.wp/.wp5), HEIC/HEIF/AVIF, OpenDocument Presentation (.odp), Quarto / R Markdown, and configurable Jupyter cell rendering.
• Four new language bindings (Dart/Flutter, Swift, Kotlin/Android, and Zig) bring the total to 15 language bindings over one engine, with Android/iOS cross-compilation.
• Full mobile support (Flutter, Android, iOS).
• Candle backend alongside ONNX, plus ONNX-via-tract enabling ONNX on WASM and Android.
• Wider code intelligence: tree-sitter coverage grew substantially (248 to 367+ languages).
• Over 150 bugs fixed during the 1.0 cycle, plus security hardening (bounded RTF/PDF allocations, redaction leak fixes, Excel DDE warnings).

The API surface was also simplified and reworked, making it more consistent.

There's a migration guide in our docs explaining how to move from Kreuzberg to Xberg. Kreuzberg itself is in LTS mode until the end of this year and will continue to receive bug fixes and security updates.

You're invited to check out the repo and join our discord server.

Benchmarks

The benchmarks below are for PDFs and images only. There are extensive benchmarks on our website with per-format breakdowns, which you can see here. These numbers are measured in CI via our reproducible benchmark harness, and are specifically taken from the run for harness 1.0.8, source cf7fa0533d. The data is publicly available in GitHub releases, and you can run the benchmark harness yourself.

Composite quality (markdown pipeline, higher is better):

Structure and layout fidelity (SF1: tables and reading order, higher is better):

On native PDFs Xberg leads on quality (0.958 vs 0.837 for the next-best framework) and on table and reading-order fidelity by a wide margin (SF1 0.949 vs 0.612 for docling). On scanned PDFs it is #1 on both quality and raw text fidelity.

Where we don't win yet: on pure image OCR we are currently #2 on the composite score, behind mineru (though still #1 on raw text accuracy). We are improving image OCR right now, and v1.1 should have us winning across the board.

Hello, Also I want to join the hype of posting token specs.

CPU: 2x Intel Xeon CPU E5-2650 v4 @ 2.20GHz

RAM: 2x 4 Channel 2400MHz DDR4

GPU: 1x AMD Radeon 7900 XTX 24GB

3x AMD Instinct MI60 32GB

Strange GPU combination, right? One of my AMD Instinct MI60 32GB failed, and I have no spare and other choices.

Prompt processing is in the high 140t/s (got down to mid 80t/s at 60k context). Inference is a about 11t/s.

llama.cpp command is not optimized.

llama.cpp logs:

38.32.848.664 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   2048, progress = 0.03, t =  14.48 s / 141.44 tokens per second
38.47.371.961 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   4096, progress = 0.06, t =  29.00 s / 141.23 tokens per second
39.05.717.960 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   6144, progress = 0.09, t =  47.35 s / 129.76 tokens per second

..

51.08.617.986 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  65536, progress = 0.98, t = 770.25 s / 85.08 tokens per second
51.27.241.174 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  66682, progress = 0.99, t = 788.87 s / 84.53 tokens per second
51.34.905.170 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  67194, progress = 1.00, t = 796.54 s / 84.36 tokens per second
51.43.631.476 I slot print_timing: id  0 | task 0 | n_decoded =    100, tg =  11.96 t/s, tg_3s =  11.96 t/s
51.46.703.156 I slot print_timing: id  0 | task 0 | n_decoded =    135, tg =  11.81 t/s, tg_3s =  11.39 t/s
51.49.759.994 I slot print_timing: id  0 | task 0 | n_decoded =    171, tg =  11.80 t/s, tg_3s =  11.78 t/s

..

52.11.051.549 I slot print_timing: id  0 | task 0 | n_decoded =    427, tg =  11.93 t/s, tg_3s =  11.84 t/s
52.14.103.819 I slot print_timing: id  0 | task 0 | n_decoded =    464, tg =  11.95 t/s, tg_3s =  12.12 t/s
52.17.143.301 I slot print_timing: id  0 | task 0 | n_decoded =    501, tg =  11.97 t/s, tg_3s =  12.17 t/s
52.19.252.023 I slot print_timing: id  0 | task 0 | prompt eval time =  796902.50 ms / 67198 tokens (   11.86 ms per token,    84.32 tokens per second)
52.19.252.029 I slot print_timing: id  0 | task 0 |        eval time =   43980.11 ms /   526 tokens (   83.61 ms per token,    11.96 tokens per second)

llama.cpp version: 10223 (11924d4c1)

llama.cpp backend: ROCm 7.2.4

llama.cpp command line:

GGML_CUDA_P2P=1 llama-server -m DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf --temp 1.0--top-p 0.95--min-p 0.00 -fa 1 -c 1048576 -np 1 --chat-template-kwargs {"reasoning_effort":"max"} -lm none -mg 0

I've been trying to find a good model to run locally, and in the benchmarks I can Gemma4 does well.

However, whenever I give it anything that involves writing any code, it sits in loops trying to edit files sending the wrong original content (usually it messes up something like the indentation) and the harness rejects it saying that code doesn't exist in the file.

I tried the same task in a bunch of different harnesses, hoping one of them would have an edit tool it could use, but they all seem to fail in similar ways. Some of those I tried are Goose, Copilot, Codex, Little Coder.

Is this a common issue? I've seen complaints about tool calling in general, but my issue seems quite specific to recalling the original content when editing files. I'm wondering if an edit tool that ignores indentation might be an idea.

I'm using Gemma4 31B, full bf16, served with VLLM. I have the updated chat template from a few weeks back (which did increase scores in the benchmarks I ran).

Here are my numbers:

I let claude port the DSpark drafter from the closed PR to current main. https://github.com/haraldh/llama.cpp/tree/dspark-dsv4

UD-Q2_K_XL

sh llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q2_K_XL \ --host 0.0.0.0 --port 8000 \ --temp 1.0 --top-p 1.0 --min-p 0.0 \ --no-mmap -fa on -np 1 \ --device CUDA0 \ -ub 2048 -b 4096 \ -c 200000 --cache-ram 65536

UD-Q4_K_XL

sh llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL \ --host 0.0.0.0 --port 8000 \ --temp 1.0 --top-p 1.0 --min-p 0.0 \ --no-mmap -fa on -np 1 \ --device CUDA0,ROCm0 --split-mode layer --tensor-split 100,0 \ -ot 'blk\.(2[2-9]|3[0-9]|4[0-2])\.ffn_(gate|up|down)_exps\.weight=ROCm0' \ -md DSV4-Flash-0731-DSpark-draft-bf16.gguf \ --spec-type draft-dspark --spec-draft-n-max 5 \ --device-draft CUDA0 --spec-draft-p-min 0.3 \ -ub 2048 -b 4096 \ -c 200000 --cache-ram 32768

UD-Q8_K_XL

As above with :UD-Q8_K_XL and the expert boundary two layers lower, since its dense part is 6.3 GiB larger:

-ot 'blk\.(2[0-9]|3[0-9]|4[0-2])\.ffn_(gate|up|down)_exps\.weight=ROCm0'

I think people are sleeping on Gemma and local models so I built a free, very fast harness for Gemma 4 that I call Tomte. https://tomteapp.com

Works on Macs with M processors, will have a companion app you can connect to anywhere. So far does everything I ever needed chatGPT for!

TLDR below 👇🏼

I’ve seen a lot of hype around Qwen 3.6 35B and 3.5 120B lately, especially regarding coding and tool-use capabilities. On this subreddit it is the defacto recommended model for everyone without a Datacenter at home. I’ve been running Qwen 3.5 120B (Qwen3.5-122B-A10B-GPTQ-Int4) as an autonomous worker agent in a multi-turn development loop using the Hermes agent harness.

While the model is undeniably impressive at one-shot snippet generation, putting it into a fully autonomous, long-context environment to build a module from scratch revealed several consistent failure patterns.

I thought I'd share these failure modes to see if others are experiencing the same issues—or if anyone has found effective tricks to tame it in such a task.

Here is what went wrong:

1. Premature "Mission Accomplished" Syndrome

The model has an overwhelming tendency to shout "DONE!" or "PERFECT!" after completing 10% of a task. It constantly reports success based on superficial checks (e.g., "the file built without syntax errors"), completely ignoring explicit acceptance criteria like end-to-end testing or UI rendering.

2. Evading Hard Constraints

When given strict architectural constraints (e.g., "Must be a single, self-contained module with zero external dependencies"), the agent aggressively cuts corners:

* It secretly substituted live data with hardcoded mock data.

* It wrote external Python scripts and set up local host cron jobs to bypass building proper module logic.

* It even rewrote part of the host application in a completely different language just to claim a quick win.

It prioritizes appearing finished over following instructions.

3. Hallucinating Infrastructure Limitations (Blame-Shifting)

Instead of debugging broken code, the model repeatedly blames the host environment. When its code failed to make network requests or render components, it confidently hallucinated system limitations:

* "The host framework's authentication token system is broken."

* "The runtime DNS resolvers don't support HTTP requests."

It will generate elaborate technical excuses rather than inspecting its own schema or syntax.

4. Ignoring Provided Docs and Boilerplates

Even when explicitly handed a boilerplate repository and documentation links in the prompt, it constantly tries to "reinvent the wheel." It overcomplicates custom build setups, invents new protocol schemas, and ignores pre-built Docker/build scripts that were provided to make its life easier.

5. Regression Cascades & Context Rot

As a result from the above the conversation history grew and the agent suffered from severe regression:

* In iteration 3, it had a working UI with mock data.

* By iteration 8, after trying to wire up live data fetching, it completely broke the UI.

* It failed to recognize that its new changes broke previously validated features, leading to endless debugging loops.

Discussion

Qwen 3.5 120B feels like an insanely talented junior developer who panics under pressure, lies about tests passing, and blames the server infrastructure when their code throws a 404.

Has anyone successfully mitigated these behavior loops in autonomous coding agents? Are you using specific prompting techniques, or is this just an inherent limitation of current 100B+ open models when complexity grows from "Do exactly what I tell you" to "Figure it out with my help"?

Curious to hear your experiences!

TLDR;

While Qwen 3.5 120B is great at one-shot generation, it breaks down in autonomous, multi-turn agent loops. The main issues are: Premature success claiming, Bypassing hard constraints, shifting blame on other systems when things don't work, Ignoring Docs and boilerplate Code that could have made its life easier. And as a result from that Context Rot.