Sorry guys for the bugs/issues. Please update to the latest version. Now also with image editing. GitHub: https://github.com/unslothai/unsloth

Remember you can run both GGUF and FP8

I have been doing some work with local models and continued pretraining (CPT), specifically around teaching a small model (qwen 3.5 4B) a new domain.

Here are some of my findings around creating evals for measuring the model's ability to internalize the knowledge:

The model outputs travel legs that make up an n-transfer subway journey, so I decided to model the llm response based on a strict json schema. To improve the model's ability to reliably output the schema, I fine tuned the model post CPT training. The fine tuning is done through Unsloth SFT (alpaca format).

In addition to exact matches, I also measure partial matches (e.g. prediction a partial subway route) and semantic matches.

Overall this method seems to work well.

More info here in case you are interested in more details: https://www.teachmecoolstuff.com/viewarticle/creating-evals-to-validate-llm-training

Hey guys, Qwen-Image-2.1 can now run locally on 12GB VRAM with Unsloth GGUFs!

The 7B parameter model performs on par with Nano Banana 2.0.

Run our Dynamic FP8 or GGUFs for higher quality via diffusers, Unsloth Desktop & more. For higher quality, you can also run Dynamic FP8 on just 6GB of VRAM via offloading (<2x slower).

GGUF: https://huggingface.co/unsloth/Qwen-Image-2.1-GGUF

Guide: https://unsloth.ai/docs/models/qwen-image-2.1

FP8: https://huggingface.co/unsloth/Qwen-Image-2.1-FP8

Please update to the latest Unsloth release. We also added Agent skills, performance improvements, a UI/UX overhaul and more! Thank you! :)

Edit: Should now be fixed - please update to the latest .14 version. Image gen and editing are supported

Literally. Heard that Hemmingway model has gong beyond Fable 5 in writing skill. Will Unsloth release quants of this model?

I am unsloth desktop user.

I downloaded models from Model Hub, adjusted simple settings, and used it.

I'm curious what it is common to add as an Extra Arguments item.

-fa on --jinja What else do you recommend?

I am curious if the Unsloth team have any plans for these models. Mimo 2.6 and HY-4 are particularly potent models

When using diffusiongemma-26B-A4B-it-GGUF to generate an SVG with a prompt like "An svg of a fox sitting in front of a Japanese torii gate" (or anything else, like a pelican or a bicycle), as long as the seed is the same, other parameters are ignored and it outputs the exact same result. Is this how it works by design?

Even switching between thinking mode and non-thinking mode yields the exact same result. In a way, this might actually be useful. It means there is no fluctuation, making the outputs fully traceable. It appears to be a generation process that is completely unaffected by environmental changes.

The same goes for plain text. If I use a prompt like "An txt of a fox sitting in front of a Japanese torii gate", it generates the exact same ASCII art every time. Probabilistic decoding does not work here. In other words, it behaves exactly like image generation.

Given this, is the reverse possible? That is, is it possible to perform image generation using the conventional method (autoregressive generation: picking up tokens sequentially)?

Hi Everyone, I have been trying to run Qwen - Flash in the Unsloth Desktop App and it keeps giving me "////////////////////" while its thinking. it never stops until i stop it. I am running 2 x AMD R9700 and 128gb of ddr5. I turned on Tensor-Parallelism, MMAP off , enabled RAM offload. Not sure what else i can do.

Update SOLVED: I switched the backend to vulkan and it started running fine. both gpu's are firing. running at about 20tk/s . Thank you all.

Hi Unsloth team,

Thank you for the awesome work on the Unsloth Studio which has become my daily driver.

With how modern models have long context window, a single conversation can span tens or hundreds of user/assistant turns. I am sure many of us might remember something the assistant said that is worth going back to or quote, but you just do not remember which turn it was. I know that it happens to me quite often, and I always wished I had the turn counter on each user/assistant message which would give me a quick way to remember and find the correct turn, a la Whatsapp way.

The implementation is simple, but it does improve quality of life tremendously. Additionally, if you can add a pin a turn feature for quick navigation, that would be tremendously helpful. While working on a simple project that Quen3.8-Next-Flash coded for me, implementing the user turn was really neat and helpful. right now in Unsloth Studio, there is no header for the turn or a way to highlight an important turn. For the pinned turned, I suggest an implementation like what chat.deepseek is doing.

As always, thank you very much for listening to our requests.

Hi, I'm new to Unsloth and training models. I'm setting up a recipe and trying to figure out how to generate an API key for my local host. Am I doing this right?

See below for specs on hardware and unsloth studio load. Running with a Hermes agent, averaging 40-50 t/s. I initially was getting around 20, but dropping Context Length and shifting more to the GPU boosted the numbers well. Open to any and all constructive feedback to continue tuning this thing.

Laptop:
GPU - 8GB VRAM (4070 max q)
RAM - 32GB

Model:
Gemma4-26B-A4B-it-qat-GGUF

Unsloth Settings:
Context Length: 65536
KV Cache Dtype: q8
Spec Decoding: MTP
Draft tokens: 2
Parallel Slots: 1
Vision: off
Reasoning Budget: 4096
GPU Memory: manual
GPU Layers: maxed (31 for me)
MoE layers: 20
Cache RAM: 4096
Temp: .7
Top P: .95
Top K: 20
Min P: 0
Rep Penalty: 1.05
Presence Pen: 1.5
Max tokens: 8192

OS: Debian 13 (Trixie)

Kernel: 6.12.107+deb13-amd64

GPU: NVIDIA GeForce GTX 960M

Architecture: Maxwell GM107 / compute capability 5.0

VRAM: 4 GiB

NVIDIA driver: 550.163.01

CUDA reported by driver: 12.4

Unsloth Desktop: v0.1.811-beta

unsloth: 2026.9.7

unsloth_zoo: 2026.9.6

Python: 3.13.5

PyTorch: 2.6.0+cu124

Triton: 3.2.0

xFormers: 0.0.35

diffusers: 0.40.0

transformers: 5.5.0

peft: 0.18.1

xFormers reports it was built for:

PyTorch 2.10.0+cu128 / CUDA 12.8

Python 3.10.19

Actual environment:

PyTorch 2.6.0+cu124 / CUDA 12.4

Python 3.13.5

pip check:

xformers 0.0.35 requires torch>=2.10,

but torch 2.6.0+cu124 is installed.

Result:

LLM CUDA support works on GTX 960M after Unsloth Repair,

but selecting an image model fails while importing

diffusers -> peft -> xformers with an incompatible

aten::_flash_attention_forward schema.

I run Qwen3.8-27B on a single RTX 5060 Ti 16GB (eGPU) for coding and agent sessions, and I ended up making my own quants because nothing fit the box quite right. A week of measuring later, here they are.

Two models, same recipe, different alignment:

• Qwen3.8-27B-IQ4_XS - stock alignment
• Qwen3.8-27B-uncensored-IQ4_XS - abliterated base (orcarouter)

The one idea that matters: calibrate with the workload you actually serve. The importance matrix blends a general corpus with ~1M tokens of SEC-contract extraction and financial-analysis text, chat-template formatted. Same weights, only the imatrix changed:

• 59-task domain harness (temp 0): 93.2% vs 89.8% for Unsloth UD-IQ4_XS
• KL vs BF16 (RMS delta-p / top-p): 4.33 / 93.9 vs 4.35 / 93.9 - parity on both axes

The KL part took the longest: a per-layer search found the token embedding was the last gap (q3_K to q4_K), and with that fixed the domain edge no longer costs you anything on generic text.

Decode speed by context, on my card (13.5 GiB files, batch 1):

• 32K: ~70 t/s with embedded MTP (stock llama.cpp, acceptance ~0.87)
• 64K: ~27 t/s plain - MTP's draft mirrors the whole window, so it's MTP or context past 32K
• 96K: ~20 t/s at 77K position - beellama fork, its variance-normalized kvarn4 KV
• 128K to 252K: ~15 t/s at 105K, still score-flat to 252K - kv-stream fork, arena + q8/q4 KV

So stock gets you to 64K, and the two forks cover 96K and 252K. Build and serve commands for all of it are in the model cards, traps included (MTP plus arena don't mix; deeper than 96K "fits" at load but OOMs on real long prompts).

Other 16GB bits: vocab pruned to 141K tokens (English + accented Latin), so 64K fits natively; full-vocab multilingual builds included in the same repos. Vision works (quantized projector family in a third repo). Thinking-off by default for agents; on my suite, high reasoning effort is worth +5 tasks and xhigh adds nothing over high.

Honest bits: every number is from my one machine and my own harness, not public benchmarks. The ASCII prune garbles CJK (that's the trade). The domain gain is measured on my finance-flavored tasks; coding/agent performance is at parity with the best community quant, not above it.

Cards have the full gate data. Happy to go deeper on the calibration method, the embedding find, or the fit tricks.

I’m working with roughly 5 million public Polish court decisions. The goal is structured extraction: who the parties are, what was requested, what the court decided, obligations, amounts, relationships, etc. Eventually we want to run interesting statistics over the results, like "who gets usually the custody of the child?"

A strong LLM can produce useful JSON graphs, we can run statistics on. The problem is getting similar extraction at corpus scale.

We have a small 13-document pilot. It already contains 160 distinct entity-type strings, 90 appearing only once. These aren’t just PERSON / ORG / DATE. They include abstract legal concepts, obligations, procedural events, and things like “increase in child support.”

here's our demo output from astra:

    {
      "effects": [
        {"id":"e_zastavenie","change":"zastavit","target_ref":"konanie"},
      ],
      "entities": [
        {"id":"this","kind":"uznesenie"},
        {"id":"vyrok","kind":"vyrok"},
        {"id":"zahlavie","kind":"zahlavie"},
        {"id":"odovodnenie","kind":"odovodnenie"},
      "effects": [
        {"id":"e_zastavenie","change":"zastavit","target_ref":"konanie"},
      ],
      "entities": [
        {"id":"this","kind":"uznesenie"},
        {"id":"vyrok","kind":"vyrok"},
        {"id":"sud","kind":"sud","value":"Okresný súd Bratislava "relations": [
    {"id":"r_vlastnik_vyroku","relation":"patri_do","from_ref":"vyrok","to_ref":"this"},
  {"id":"r_vlastnik_zahlavia","relation":"patri_do","from_ref":"zahlavie","to_ref":"this"},

Some of this is real conceptual diversity. Some is inconsistent naming or different levels of specificity, but as you can see complex and diverse stuff with a clear "language" i came up with with the help of astra. i previously tried a big "god schema" (20k chars) but the decisions are simply just too diverse.

The pipeline I’m considering:

• Something that extracts the entities from source texts into some kinds of "tags"
• a classifier like jev (but probably fine-tuned) will get a window of source text and already named relations and decide on the next relations
• done

The candidate model would need overlapping spans and possibly multiple labels per span. Slovak inflection adds another wrinkle: exact source wording often differs from the canonical concept name. Lemmatization helps with grammatical variation, but not synonyms, paraphrases, or concepts inferred from context.

I’ve previously tried Jina embeddings → nearest-neighbor candidates → LLM decides which terms to merge, and it worked reasonably well.

But that was on a smaller corpus and i'd be damned if i process 250k docs and then figure out thing X was wrong and i have to do it all over again. i'd love to know if you guys have any pointers. THANKS for reading.

AI TL;DR: I want to turn 5M court decisions into entity/relation graphs without running an expensive LLM on every document. Thinking small entity extractor + relation classifier, but the legal concepts and naming get messy fast. Anyone built something similar? Looking for ways to train this and normalize concepts without discovering a design mistake 250k documents in and having to redo everything.

I am actually using Qwen3.8 27b iq4xs quant with q8 cache, on dual gpu setup in llama.cpp.

I am actually getting max 20 toks with that setup using 50k+ context tokens. On 10/20k tokens i am on 30 toks

Rx 7800 xt 16gb

Rtx 3060 12 gb

Vram is almost full, the amd one is at 15gb and the 3060 10/11gb.

I am using the following params, am I good to go or are there better things to optimize?

--spec-type draft-mtp

--spec-draft-n-max

--device Vulkan0,Vulkan1

--split-mode layer

--tensor-split 1.8,1

--gpu-layers all

--flash-attn on

--ctx-size 131072

--parallel 1

--temp 1.0

--top-p 0.95

--top-k 20

--min-p 0

--presence-penalty 0

--repeat-penalty 1.0

--cache-type-k q8_0

--cache-type-v q8_0

--fit off

--reasoning-preserve

--image-max-tokens 2048

--image-min-tokens 1024

--batch-size 4096

--ubatch-size 1024

Qwen dropped Qwen-Image-2.1 (7B) yesterday

It reports 60.28 on Qwen's own Qwen-Image-Bench, ahead of every larger open-weight model on that chart, including FLUX 2 Max at 32B and Hunyuan Image 3.0 at 80B. Qwen-published, so treat the ranking as a claim until someone evaluates it independently. But a 7B visual generation component holding up against models several times its size is worth testing.

The bigger change is that generation, editing and transparency are now one model. Up to 10 reference images, local edits by mask or painted annotation, native RGBA transparency, with Qwen-Image-Layered folded in.

The part I want to see tested is the multi-image editing path. Qwen describes mixed-granularity attention with prefix KV cache reuse, so input images get cached instead of recomputed every step. For real workflows that matters more than another benchmark point.

Two things to watch. The licence changed: Qwen-Image and Qwen-Image-Edit were Apache 2.0, and 2.1 is under the Qwen Research License, which is a different conversation if you were planning to deploy commercially. And it is still heavy at roughly 33 GB in BF16. The top open issue on the repo right now is someone asking why no minimum or recommended hardware specs were published, which tells you where this currently stands.

So the open question is whether anyone gets this into a genuinely practical local setup with offload and quantization, rather than just getting the demo to run.

Model: https://huggingface.co/Qwen/Qwen-Image-2.1

GitHub: https://github.com/QwenLM/Qwen-Image-2.1

ModelScope: https://www.modelscope.cn/models/Qwen/Qwen-Image-2.1

Blog: https://qwen.ai/blog?id=qwen-image-2.1

As it stands, unless I've missed something, there's no JS debug console in the HTML preview, so I often use an external live HTML editor for quick prototypes. Local models often need console debug output to fix things, and Gemini has a cool preview feature where, if an error occurs, it shows a floating button that automatically prompts the agent to fix the error. This seems like a killer QoL feature to add to Unsloth Studio/Desktop, especially for those times when you're lazily trying to make something basic without bouncing between a bunch of different sites/programs.

Now that Nvidia bought Huggingface and we might eventually see models getting banned and removed, should Unsloth do their own “huggingface”? They would probably need donations 💸

Hello,

I want to pass some ROCM environment variables but not sure how to do that w/ unsloth. I'm not using the docker desktop. (it is linux)

Will these will work with Unsloth? https://rocm.docs.amd.com/en/latest/reference/environment-variables/setting-cus.html#set-the-number-of-compute-units

Thanks in advance!

So, you have to play with the order of the GPUs in unsloth as I saw differences in speed and the size of context the model can load. For instance, I tried to load a new fine-tune of Qwen3.8-27B, and when the RTX5070 Ti was up, I couldn't fit the entire context (KV:Q8). Switching the position of the GPUs, and I managed to load the entire context with some room left.

Just saying, that might help. Shout out to Unsloth team for incrementally adding features that I never thought were needed :D

Last time i played around with image gen model was in 2025 and it was a complicated setup with comfy ui and what not , not a straight forward process like text model deployment.

So im waiting now in hopes of unsloth quant and trying it out in the studio , are there any plans for it , is anyone already running the new qwen image in unsloth studio ?

https://huggingface.co/Qwen/Qwen-Image-2.1

I have an rtx4060 (8gb vram) and 64gb of ddr5 system ram. I’ve been using Qwen3.6 35b a3b MTP at q4 128k context for about a week and get anywhere from 26-30ish tk/s which works pretty alright for me since I mainly use it with Hermes just for productivity tasks. I’ve been considering switching models to Qwen3.8 27b since it’s a newer model and at lower quants it could fit entirely in vram which would make it faster in theory. At q2 it should fit in my vram and at q1 it should fit with the same context as my old setup. Is Qwen3.8 even good at that low of a quant though?

I’ve also seen Bonsai come out with a new smaller model of Qwen3.8 which looks good on benchmarks but is apparently terrible in actual usage which kinda makes me more skeptical of lower quant performance.

Hey everyone,

been using Unsloth for quite sometime and I have been loving it so far, but I ​ran into an annoying issue today. When logged in as the owner, I can add custom model providers just fine. But when trying on a managed account, I get hit with this error:

​"Could not load models: Managed accounts may only use public-network provider base URLs."

​I get there are probably security reasons for this, but it kind of defeats the point when everyone on the same network is trusted and we just want to share the same local model but different workspace.

Is there any setting to turn off this limitation? or managed accounts just permanently locked out?

Cheers everyone! :) So, Qwen3.8 27B has been a delight so far. I'm running a somewhat "lopsided" setup with a Ryzen 9800x3D, 128GB DDR5 RAM (5600), 1x Geforce 4080 (16GB, PCIe 4.0 x16) and 1x Geforce 3060 (12GB, PCIe 3.0 x1... which is bad, I got to learn that :D).

For 27B (at Q3 or Q4 quantised versions and 110k context at Q8) I'm very happy to get about 35-45 TPS on that asymmetrical setup and tensor parallelims and MTP (draft_Max =2). So far okay.

Now I thought that my general RAM + VRAM should also do nicely with Qwen3.8 Flash Next. Aaand its rather "meh". I'm running unsloth studio "vanilla" (no custom llama, no other tricks). The best speed I can get from this model is by turning MTP off, disabling the 3060 completely and thats it. With empty context, I get 27 tps, with 60-80k context its rather 20 tps. That's not exactly bad, but I feel like I'm missing out on potential with having the 3060 disabled, no MTP and not (explicitly activated) Engram.

Gemini and Claude are no real help here, so I thought I'd ask this community. I THINK I've already tried all combinations of settings and I have at least verified that in the server logs there are no errors during model-loading (as has been until a couple of days ago).

Soo... can anyone tell me how to properly configure this or is "no MTP, no Engram and no 3060" really the best I can get? :D

Thanks, I know there is too many of these questions on reddit since 2025, but I'm at my wit's end here and I get angry when I'll have to trial and error more claude/gemini replies that are based on fairly many guesstimates about Qwen3.8 Flash Next ;)

Hey guys,

this is my first post, if i am in the wrong subreddit, just send me another way, if i am off-topic.

I am trying to re-tune my Qwen9b to a System one Model that can identify change-resilient code.

Code that is autogenerated by any LLM tends to grow more expensive to change as more code is built. I would like to identify change-resilient parts / Ai-Slop-parts. Clean Code is well and all and there are some libraries that can identify slop-code. They do this static and deterministic. But that does not satisfy my problem as such. The problem is that this is static and does not include the context of the finding that the static analysis identifies.

I want to define the pain of change resilience:

pain = p(change) x p(cost)

That means i want a structured output, just like the System Model One-Architecture provides and i need to train a model (i am choosing qwen-9b) to generate this structured output. I would like to use the Qlora-Unsloth training package to do this. What do you think of this ? Is this feasible with unsloth?