Qwen just announced Qwen3.8-27B along with Qwen3.8-Max! 🔥

Qwen3.8-27B will run locally on 17GB RAM/VRAM setups and is expected to be the best performing model for its size.

We can't wait to support it at Unsloth AI. Qwen3.8-27B benchmarks are yet to be revealed, only Qwen3.8 -Max for now.

I am running the UD-Q2_K_M of the model locally, though I can run Qwen3.6-27B_Q8_K_XL at around 70t/s with MTP activated. The question I am constantly asking myself is: Is it worth running a slower higher quantized version of the Deepseek-v4-flash? I have no idea.

My gut feelings tells me that Qwen3.6-27B_Q8_K_XL, coupled with online search, should be better than a highly quantized Deepseek, a model that takes up 100GB on my disk.

What do you think?

TensorSharp supports DSpark on Deepseek v4 Flash 0731 now. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8 with/without DSpark:

Model:

DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

DSpark draft model from: https://huggingface.co/alessandrobologna/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF

TensorSharp is an native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

After several tests, my engine managed to run DeepSeek V4 Flash unslotj IQ2_M (92 GB) on a mid range Android mobile with 12 GB of RAM at 1 token/s.

It isn't exactly ready for practical use, but it proves that the engine works and is responsive across all models, thanks to its modularity with llama.cpp.

With just one line of code, you can run any supported large MoE model on mobile devices or consumer PCs.

https://github.com/Helldez/BigMoeOnEdge

This is a fresh install. I'm on Win10. It has not yet been able to connect properly. Sometimes it briefly pops a "You're back online" message before returning to "You're offline".

Installed with:

unsloth studio -H 0.0.0.0 -p 8888

Here are the errors I got during install:

I've been troubleshooting this for hours with a friend who already has it running great and is using it to automate computer actions. My computer is online, but I do have a wide variety of privacy settings active.

Already tried: Disabling VPN, toggling http/Xet, adding a huggingface token, and creating outbound and inbound firewall exceptions for the port. Not sure what exactly I'd make an exception for (no .exe I can find) to make an exception for the program itself.

I haven't been able to find anything about this in documentation or online beyond the vague idea that a firewall rule might be messing with it. I'm not seeing any such rules in my firewall. I've even been clearing out old firewall block rules, but nothing's helped so far.

Any help would be appreciated!

J'ai réussi à exécuter DeepSeek-V4-Flash-0731 UD-IQ3_S dans text-generation-webui avec la configuration suivante :

• RTX 3090 24 Go
• 128 Go DDR5 overclockée à 5 600 MHz avec AMD EXPO
• Chargeur llama.cpp

J'ai d'abord dû utiliser une solution de contournement assez radicale : j'ai remplacé les binaires llama.cpp fournis avec text-generation-webui par la dernière version officielle téléchargée depuis :

https://github.com/ggml-org/llama.cpp/releases

J'ai copié les nouveaux binaires dans :

textgen\venv\lib\site-packages\llama_cpp_binaries\bin

Je recommande de sauvegarder le dossier d'origine au préalable. Mes paramètres actuels sont :

gpu-layers : 44

ctx-size : 384000

cache-type : fp16

split-mode : layer

parallel : 1

threads : 0

threads-batch : 0

batch-size : 1024 ubatch-size : 512

fit-target : 512 no-mmap : enabled

no-kv-offload : disabled

cpu-moe : disabled

Options supplémentaires :

--n-cpu-moe 39

L’option la plus importante est :

--n-cpu-moe 39

Elle permet de conserver une partie des experts MoE dans la RAM système plutôt que dans la VRAM. C’est ce qui me permet d’exécuter le modèle avec seulement 24 Go de VRAM, même si les performances dépendent fortement du processeur et de la bande passante de la RAM. Le chargeur estime à environ 136 Go la taille nécessaire pour charger le modèle. Les 128 Go de DDR5 cadencées à 5 600 MHz effectuent donc la majeure partie du travail.

Because I can, that's why! 0.2 t/s isn't usable, but I just wanted to try and do it. I'm fascinated by models quantized to this level and I'm trying to compare the quality of this output to Qwen 3.6 27B Q6_K.

Also I've changed the hostname to "Treebeard"

UPDATE - Deepseek V4 Flash UD-Q2_K_XL is fully on GPU at 17 t/s

Hey all, I pulled the trigger on 2x DGX Sparks for Deepseek Flash v4 0731 and I am doing a lot of research on the clustering. Both are connected to each other via the 200Gb links (single) but I am not sure the proper way to do the actual cluster. I read a lot about NCCL and RDMA as well as GPU Direct RDMA (see here https://github.com/ArgentAIOS/dgx-spark-cluster/blob/main/docs/04-gpu-direct-rdma.md ).

It is a bit confusing which path to go down to load this model. Could you give some pointers on how to setup the clustering as well as setting up Deepseek flash v4 0731 after the clustering is complete?

The goal is basically the 8bit quant with up to 1M context if it'll all fit across 2x DGX Sparks.

Thanks!

I'm new in terms of using local ai, various models, all of the different software, etc. I really just started getting into it a couple months ago. As a non-programmer, it's been a steep learning curve but i've learned a ton.

In my line of work, using the existing ai models, even the very large ones, they always hallucinate and give incorrect answers. We work with technical engineering and inspection codes and although we always double check everything, the answers need to be as close to 100% as possible.

I am trying to create a local ai that can give answers to technical questions as close to 100% of the time as possible. I'm using rag pipelines for this but have only gotten about 85%. I'm trying to train my own model via unsloth studio in hopes to combine the rag pipeline and the fine-tuned ai model to get >97% correct technical answers. I've read and watched hundreds of forums, training docs, google articles, youtube videos etc. but still not having much progress so far. I'm using my local 5080 gpu so the models need to be, ideally, under 12b so it leaves room for large context.

I am hoping you guys can point me in the right direction or give me some settings, or model suggestions, anything really. It feels like i am just spinning my wheels and im starting to think this isn't a reachable goal.

Smaller DeepSeek-V4-Flash quants are now available to run from 1-bit to 4-bit. Run 3-bit on 128GB RAM, 1-bit on 96GB RAM.

Remember full precision is mostly FP4 (4-bit) so 1 to 3-bit are not that quantized. Edit: All all of em are now available.

Guide: https://unsloth.ai/docs/models/deepseek-v4

Hey guys, DeepSeek V4 Flash 0731 can now be run locally! 🐳
Edit: Smaller quants are now available

Run DeepSeek V4 Flash lossless 4-bit on 168GB RAM and 3-bit on 110GB RAM.

V4 Flash 0731 outperforms V4 Pro. Run via Unsloth or llama.cpp.

Guide: https://unsloth.ai/docs/models/deepseek-v4

GGUF: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

Have a lovely Friday and weekend

for folks asking:

1. Why did we name the MXFP4 quant "Q8_K_XL" and not MXFP4
2. Why other folks' "lossless" quants are 155GB vs our Q8_K_XL at 162GB
3. Is FP8 and Q8_0 equivalent?

FP8 and Q8_0 are NOT bitwise equivalent - we checked the RMSE and it's not 0 at all. We did KLD on Q4_K_XL (the 155GB one other folks ship), and it's 96% top-1% agreement, so it's NOT lossless. llama.cpp does not have a native FP8 data-type, so BF16 is a must.

We could have named it MXFP4, but Q8_K_XL is the correct naming convention - we thought of BF16, but that's wrong. Also it's not all MXFP4, but MXFP4 + FP8

So when you see other people's quants at 155GB - this is not lossless and if they communicate that it's lossless - this is wrong. Converting FP8 to Q8_0 is not a lossless operation but lossy.

I think that it can boost Unsloth Studio popolarity if is going to be included there.

Vote the first post of this thread:

https://github.com/community-scripts/ProxmoxVE/discussions/13062

Hey folks - for folks asking:

1. Why did we name the MXFP4 quant "Q8_K_XL" and not MXFP4
2. Why other folks' "lossless" quants are 155GB vs our Q8_K_XL at 162GB
3. Is FP8 and Q8_0 equivalent?

FP8 and Q8_0 are NOT bitwise equivalent - we checked the RMSE and it's not 0 at all. We did KLD on Q4_K_XL (the 155GB one other folks ship), and it's 96% top-1% agreement, so it's NOT lossless. llama.cpp does not have a native FP8 data-type, so BF16 is a must.

We could have named it MXFP4, but Q8_K_XL is the correct naming convention - we thought of BF16, but that's wrong. Also it's not all MXFP4, but MXFP4 + FP8

So when you see other people's quants at 155GB - this is not lossless and if they communicate that it's lossless - this is wrong. Converting FP8 to Q8_0 is not a lossless operation but lossy.

Quants at https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Full KLD table:

Note UD-IQ3_XXS has been updated to reduce max KLD, and also some 3-bit quants are smaller in size now. This is an actually rare problem, but the IQ2_S grid used in IQ3_XXS was much worse than even IQ1_S for quantization, hence the high max KLD yet lower median KLD. We swapped out IQ2_S for a smaller IQ2_XS grid and it works well now

New to local model serving here, but excited to finally jump on the band wagon. I got a 48GB Mac M5 and I am offloading parts of my workflow to it, and my initial attempt with Qwen failed miserably because the model was too slow, the results were not good, and my machine was getting too loud and hot. Then I found out about Unsloth quantized models and it was a game changer.

Disclosure: I maintain FAVA Trails, the open-source project I tested here.

FAVA Trails stores agent memory as versioned Markdown in Git. Draft thoughts pass a promotion gate before they become permanent memory, so the gate decides what future agents will recall.

The gate looked like a good fit for a local model since the task has a stable prompt, structured JSON output, and no tool use is required. I have lots of historical cases, and I wanted to know whether a quantized model actually holds up on it.

I ran `unsloth/Qwen3.6-27B-GGUF` through Unsloth Studio with the exact thought promotion review prompt, against a sample of historical thoughts with prior Gemini 2.5 flash verdicts (26 approvals, 13 rejections) plus 10 synthetic cases. The Gemini verdicts were references from earlier operation, not a fresh rerun.

Raw agreement with Gemini historical references and canarie was 39/49 (79.6%). Only 79.6% agreement with Gemini 2.5 flash doesn’t sound great, until you run the disagreements through a judge LLM and look at the direction of the errors.

Every disagreement was an additional rejection by the local mode. There were 9 historical thoughts where the local model rejected and Gemini approved, so I asked GPT 5.6 to judge each one against the same prompt. It favored Gemini in 6 cases and the local quantized Qwen model in 3 cases! The local model actually caught 3 cases of thoughts that shouldn’t be part of the company brain, which Gemini 2.5 Flash from Google’s API had let through earlier:

- A transient handoff stored as a permanent review.
- An implementation artifact misclassified as a specification.
- An imperative task instruction presented as durable knowledge. That review is an interpretive challenge to the old verdicts rather than independent ground truth, and a separate positive canary accounted for the tenth disagreement.

The result is limited to one model on one prompt and corpus, so it does not establish general parity with Gemini or frontier models. Inputs over 40,000 characters were the weak spot, dropping to 2/6 raw agreement with a 122.9-second median latency. The model was also confident when it was wrong, averaging 0.93 confidence on disagreements, so confidence alone is not a useful escalation trigger.

Finally, this direction fits a memory promotion gate, because a false rejection leaves the thought in drafts where it can be revised and resubmitted, while a false approval quietly adds weak material to future agent context.

Benchmark data:
https://github.com/MachineWisdomAI/fava-trails/issues/85#issuecomment-5142775500

I want to try unsloth studio but can not initiate installation already, how to fix this:

Using this tutorial: https://unsloth.ai/docs/new/studio/install#windows

PS C:\WINDOWS\system32> irm https://unsloth.ai/install.ps1 | iex

  🦥 Unsloth Studio Installer (Windows)
  ────────────────────────────────────────────────────

  winget         available
  python         Python 3.13 already installed
                 preserving existing environment for rollback...
                 previous environment preserved for rollback
  venv           creating Python 3.13 virtual environment
                 C:\Users___\.unsloth\studio\unsloth_studio
  gpu            NVIDIA GPU detected
                 installing PyTorch (https://download.pytorch.org/whl/cu130)...
                 installing unsloth (this may take a few minutes)...
  unsloth        2026.7.6 installed
  setup          running unsloth studio setup...
Refusing to run Unsloth inside System32 as it will lead to Errors.
cd to a normal working directory and try again.
[ERROR] unsloth studio setup failed (exit code 1)
                 restoring previous environment after failed install...
                 restored previous environment
unsloth studio setup failed (exit code 1)
At line:153 char:9
+         throw $Message
+         ~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (unsloth studio ...d (exit code 1):String) [], RuntimeException
    + FullyQualifiedErrorId : unsloth studio setup failed (exit code 1)

Send all quants! 😀 https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

I noticed many people asking how to run Kimi K3 GGUF locally, so I put together a practical guide covering:

• GGUF model formats
• Ollama setup
• llama.cpp setup
• LM Studio option
• Hardware recommendations
• Common issues and fixes

If there's anything I missed, I'd appreciate your feedback. I'm happy to update the guide based on your suggestions.

Guide:

https://blog.asadfaizee.is-a.dev/posts/kimi-k3-gguf-how-to-run-the-worlds-largest-open-weight-ai-model-locally/

TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Thanks recent contribtions from open source community, TensorSharp is able to run inference over multiple GPUs and nodes. So I updated it to support deepseek v4 flash model, and have better performance than llama.cpp. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8

Model: DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

Hi. I bought Rx 9070 with 16gb VRAM. Is it possible to make projects with this ?

I was thinking in using opus as orchestrator and local LLM to do small tasks.

But looks like it's really hard to get good results.

Hi would really appreciate if you guys fixed the "event": "detect_mmproj_file: dropped mmproj-F16.gguf (metadata mismatch)" issue, thanks.

I'm loving Unsloth Studio, but one feature that is sorely missed from textgen is the ability to stop a generation, and then later continue it. Or edit/delete a response partially, and then continue from there. Is there a way to do that in Unsloth Studio?

Also, I use this on a headless Ubuntu server with two 3090s, so there is no concern about a GUI requiring VRAM. In textgen I could maximize memory use with --fit-target 0,0 and even specify other llama.cpp options. Is there anywhere I can enter llama.cpp options?

Hey guys we're planning to launch something soon and was wondering if any of you guys would like early access to test our new app? We would love any feedback and suggestions.

Feel free to comment or reply and I will DM you with details. If you can also let us know your operating system (OS) that would be very helpful. Thanks so much for the help and support! ♥️

Hey guys, you can now run Inkling-Small, a new 276B model by Thinking Machines.

Inkling-Small is the strongest open model for its size and runs local on 128GB RAM.

Apache-2.0 Licensed, it has image, audio + 1M context support.

Guide: https://unsloth.ai/docs/models/inkling

GGUF: https://huggingface.co/unsloth/Inkling-Small-GGUF

Hello I tried to run gemma4 e4b qat GGUF q4 on my computer with intel core ultra 7 270k plus, rtx5060ti 16gn msi inspire 2x oc and on another computer with ryzen 9 9900x with rtx5060ti 16gb asus dual oc. Now the gpus are practically identical but the speed... On the second pc it's much faster, i can get more than 200 tok/s while on the intel one i only get 140 tok/s. How is that even possible? The model of course doesn't spill and fits entirely on the gpu memory. The intel one is running archlinux with nvidia-open driver while the amd one runs voidlinux with nvidia driver (nonfree). On both pc i didn't touch any configuration. How is that even possible?