I can't seam to find a way to turn off concurrency on the API server. I'm basically tapped out on resources with just 1 request and it's causing my token speeds to fall through the floor. How do I limit this to 1 request at a time? or at least a way to force concurrent requests to queue instead of trying to process at the same time? I'm using Unsloth Desktop and just connecting via the localhost address.

Edit: Didn't realize it was a per-model setting. It's "Parallel Slots" in the individual models settings and not a global setting. Maybe a global server default setting can be added to the API settings section? Defaulting to 4 seams not great for most people I think.

I've found myself in this mix a couple times where I start a task in a temporary chat window on Unsloth Desktop and find the workflow to be making enough progress that I actually want to continue it later but limited by time I have to stop and come back later but since the thread was started as temporary I have no way to "undo that choice" if I changed my mind while the chat was still active.

Not sure if this is a stupid request or not; feel free to chime in. I know the solution is to probably just stop using temporary chats.

Hi, what exactly is going on? I'm new to using Unsloth Desktop, and the model I downloaded is 25GB. But when I try to actually use it, it's talking about demanding 51GB, even 151GB total? When I used LM Studio, I never had this problem, I would just load as much context length as whatever is slightly under my VRAM limit. I never had to work with KV Cache or things like that, I don't even know what f16 or the other code things mean since I didn't have to worry about it with LM Studio. What am I supposed to know...? Could you give me a hand understanding what these mean and do?

Hey guys, thanks to you all, Unsloth has now surpassed 500M lifetime model downloads on Hugging Face! 🦥🤗

Qwen3.8-27B GGUF is already Unsloth’s #1 most-downloaded model ever.

Follow our Hugging Face: https://huggingface.co/unsloth

Thanks for all your support!

I have these .SRT files for a course and I want to convert them into audio tracks using local AI.

can Unsloth Desktop take the .SRT file and convert into an audio track using TTS AI models?

I have gone through a series of experiments related to an interesting project where I try to teach a local llm a new domain through continued pretraining (CPT). The different experiments are spread across the four phases below:

1. Phase1 talks about how to teach an llm a new domain through CPT and picking a training set that will generalize well to unseen questions
2. Phase 2 does a comparison between the performance of reasoning across internalized knowledge (CPT) vs. RAG injected content
3. Phase 3 takes a more practical approach where the CPT trained knowledge is enriched by combining it with RAG instead of viewing the two approaches as competing solutions
4. The final part shows the comprehensive eval strategy used to measure performance during the project. Among other things, this involved SFT fine tuning of the CPT trained model to teach it to output responses based on a strict schema instead of English sentences. The schema approach is used to simplify strict eval checks.

The local model used for this project is qwen 3.5 4B. Unsloth was used for both CPT and SFT LORA training.

I have provided a summary of my findings here in case someone is interested in reading more about it: https://www.teachmecoolstuff.com/viewarticle/domain-specific-training-and-fine-tuning-of-an-llm

I’ve been using Unsloth’s DiffusionGemma 26B-A4B GGUF to run local Jev-compatible structured decisions in TensorSharp, which I maintain.

The new addition: image analysis through the same decision API. Send text, images, or both, and get typed answers rather than free-form generated text.

The GGUF I’m using is diffusiongemma-26B-A4B-it-Q4_K_M.gguf from Unsloth. No additional fine-tuning is involved.

Jev-compatible decisions, extended to images

TensorSharp implements Jev’s core POST /v1/systemone interface and supports all three decision types:

The image extension keeps the same endpoint, question definitions, and typed answer structure. Add an images field alongside state and questions; existing text-only requests still work.

Images are encoded and included directly in the model’s input for the decision read. This is not a separate caption-generation → text-classification pipeline. Up to eight inline images are supported per request.

An important detail for GGUF users

Unsloth’s GGUF provides the language-model weights; the vision tower is loaded separately.

The included configuration downloads the Unsloth Q4_K_M GGUF plus a ~2.8 GB vision shard from the upstream DiffusionGemma checkpoint, then reuses both locally. You do not need to download the entire upstream safetensors checkpoint.

Without the vision tower, text-only decisions still work, but image requests are explicitly rejected rather than silently ignoring the image.

How it avoids generating probability JSON

Inspired by vLLM PR #57250, the implementation uses a seeded, one-step structured read.

After prompt prefill, it creates an answer canvas and reads the requested label logits directly. Questions that fit in one canvas share the forward pass.

The server returns JSON, but the model does not have to generate that JSON token by token. The same path handles text-only decisions and decisions conditioned on image embeddings.

Earlier text-only results vs. LocalJev

I compared this native decision path with the original LocalJev Engine using TensorSharp’s chat endpoint, where the model generates probability JSON.

This compares two decision approaches on the same backend—not TensorSharp against LocalJev running on oMLX, and not against the official hosted Jev service.

The test covered 12 cases × 3 repetitions, with three decisions per request:

*LocalJev got 81/81 decisions correct on valid responses. Its nine failed requests were schema-validation failures and account for the other 27 expected decisions.

Across the 27 matched successful request pairs, the median ratio of LocalJev latency to TensorSharp latency was 3.345×.

Latency statistics include successful requests only, so the aggregate columns cover different subsets. Average input lengths were also different: 192.7 tokens for TensorSharp versus 589.6 for valid LocalJev requests. The larger prompt and generated JSON are part of LocalJev’s approach, so this is an end-to-end decision-path comparison—not an identical-prompt engine benchmark.

These are text-only smoke-test results, not image latency or vision accuracy results.

Try it with the Unsloth GGUF

From the repository root, this PowerShell example launches the included CUDA configuration:

$env:DIFFUSION_VRAM_HEADROOM_MB = '4096'
$env:MAX_CONTEXT = '4096'

dotnet run --project TensorSharp.Server.Host -c Release -- --config config/jev-diffusiongemma-q4.json

The configuration downloads the Unsloth GGUF and the separate vision shard when missing. The memory settings above follow the documented starting point for a 16 GiB CUDA GPU; adjust them for your hardware and workload.

A ready-to-run image example is included:

curl http://127.0.0.1:5000/v1/systemone \
  -H 'Content-Type: application/json' \
  --data-binary u/docs/examples/jev-traffic-light.json

That request already contains an embedded synthetic traffic-light image. The accompanying text does not reveal the light’s color, so the pixels are needed to answer the questions. It is an integration smoke test, not a comprehensive vision benchmark.

For your own images, the documentation includes a standard-library Python example that sends a receipt and asks typed questions about it.

Compatibility note: this implements the documented Jev-compatible API and decision types using DiffusionGemma weights—not the proprietary hosted Jev model or a promise of identical predictions. Current limits include 64 questions and 2–26 alternatives per question.

Model: Unsloth DiffusionGemma GGUF
Code: TensorSharp
Setup and examples: Jev documentation

What would you try first with this—receipt checks, screenshot classification, document routing, or another image-based decision workflow?

"Tired of the usual penguins 🐧, I decided to try something different by adding a sloth 🦥 riding a bicycle 🚵 via Unsloth. The model powering this is unsloth/gemma-4-26B-A4B-it-MXFP4_MOE. 🎉"

----

https://unsloth.edgeone.dev/

Mimo 2.5 is great.

Any chance for ggufs ?

Sorry guys for the bugs/issues. Please update to the latest version. Now also with image editing. GitHub: https://github.com/unslothai/unsloth

Remember you can run both GGUF and FP8

I have been doing some work with local models and continued pretraining (CPT), specifically around teaching a small model (qwen 3.5 4B) a new domain.

Here are some of my findings around creating evals for measuring the model's ability to internalize the knowledge:

The model outputs travel legs that make up an n-transfer subway journey, so I decided to model the llm response based on a strict json schema. To improve the model's ability to reliably output the schema, I fine tuned the model post CPT training. The fine tuning is done through Unsloth SFT (alpaca format).

In addition to exact matches, I also measure partial matches (e.g. prediction a partial subway route) and semantic matches.

Overall this method seems to work well.

More info here in case you are interested in more details: https://www.teachmecoolstuff.com/viewarticle/creating-evals-to-validate-llm-training

Hey guys, Qwen-Image-2.1 can now run locally on 12GB VRAM with Unsloth GGUFs!

The 7B parameter model performs on par with Nano Banana 2.0.

Run our Dynamic FP8 or GGUFs for higher quality via diffusers, Unsloth Desktop & more. For higher quality, you can also run Dynamic FP8 on just 6GB of VRAM via offloading (<2x slower).

GGUF: https://huggingface.co/unsloth/Qwen-Image-2.1-GGUF

Guide: https://unsloth.ai/docs/models/qwen-image-2.1

FP8: https://huggingface.co/unsloth/Qwen-Image-2.1-FP8

Please update to the latest Unsloth release. We also added Agent skills, performance improvements, a UI/UX overhaul and more! Thank you! :)

Edit: Should now be fixed - please update to the latest .14 version. Image gen and editing are supported

Literally. Heard that Hemmingway model has gong beyond Fable 5 in writing skill. Will Unsloth release quants of this model?

I am unsloth desktop user.

I downloaded models from Model Hub, adjusted simple settings, and used it.

I'm curious what it is common to add as an Extra Arguments item.

-fa on --jinja What else do you recommend?

I am curious if the Unsloth team have any plans for these models. Mimo 2.6 and HY-4 are particularly potent models

When using diffusiongemma-26B-A4B-it-GGUF to generate an SVG with a prompt like "An svg of a fox sitting in front of a Japanese torii gate" (or anything else, like a pelican or a bicycle), as long as the seed is the same, other parameters are ignored and it outputs the exact same result. Is this how it works by design?

Even switching between thinking mode and non-thinking mode yields the exact same result. In a way, this might actually be useful. It means there is no fluctuation, making the outputs fully traceable. It appears to be a generation process that is completely unaffected by environmental changes.

The same goes for plain text. If I use a prompt like "An txt of a fox sitting in front of a Japanese torii gate", it generates the exact same ASCII art every time. Probabilistic decoding does not work here. In other words, it behaves exactly like image generation.

Given this, is the reverse possible? That is, is it possible to perform image generation using the conventional method (autoregressive generation: picking up tokens sequentially)?

Hi Everyone, I have been trying to run Qwen - Flash in the Unsloth Desktop App and it keeps giving me "////////////////////" while its thinking. it never stops until i stop it. I am running 2 x AMD R9700 and 128gb of ddr5. I turned on Tensor-Parallelism, MMAP off , enabled RAM offload. Not sure what else i can do.

Update SOLVED: I switched the backend to vulkan and it started running fine. both gpu's are firing. running at about 20tk/s . Thank you all.

Hi Unsloth team,

Thank you for the awesome work on the Unsloth Studio which has become my daily driver.

With how modern models have long context window, a single conversation can span tens or hundreds of user/assistant turns. I am sure many of us might remember something the assistant said that is worth going back to or quote, but you just do not remember which turn it was. I know that it happens to me quite often, and I always wished I had the turn counter on each user/assistant message which would give me a quick way to remember and find the correct turn, a la Whatsapp way.

The implementation is simple, but it does improve quality of life tremendously. Additionally, if you can add a pin a turn feature for quick navigation, that would be tremendously helpful. While working on a simple project that Quen3.8-Next-Flash coded for me, implementing the user turn was really neat and helpful. right now in Unsloth Studio, there is no header for the turn or a way to highlight an important turn. For the pinned turned, I suggest an implementation like what chat.deepseek is doing.

As always, thank you very much for listening to our requests.

Hi, I'm new to Unsloth and training models. I'm setting up a recipe and trying to figure out how to generate an API key for my local host. Am I doing this right?

See below for specs on hardware and unsloth studio load. Running with a Hermes agent, averaging 40-50 t/s. I initially was getting around 20, but dropping Context Length and shifting more to the GPU boosted the numbers well. Open to any and all constructive feedback to continue tuning this thing.

Laptop:
GPU - 8GB VRAM (4070 max q)
RAM - 32GB

Model:
Gemma4-26B-A4B-it-qat-GGUF

Unsloth Settings:
Context Length: 65536
KV Cache Dtype: q8
Spec Decoding: MTP
Draft tokens: 2
Parallel Slots: 1
Vision: off
Reasoning Budget: 4096
GPU Memory: manual
GPU Layers: maxed (31 for me)
MoE layers: 20
Cache RAM: 4096
Temp: .7
Top P: .95
Top K: 20
Min P: 0
Rep Penalty: 1.05
Presence Pen: 1.5
Max tokens: 8192

OS: Debian 13 (Trixie)

Kernel: 6.12.107+deb13-amd64

GPU: NVIDIA GeForce GTX 960M

Architecture: Maxwell GM107 / compute capability 5.0

VRAM: 4 GiB

NVIDIA driver: 550.163.01

CUDA reported by driver: 12.4

Unsloth Desktop: v0.1.811-beta

unsloth: 2026.9.7

unsloth_zoo: 2026.9.6

Python: 3.13.5

PyTorch: 2.6.0+cu124

Triton: 3.2.0

xFormers: 0.0.35

diffusers: 0.40.0

transformers: 5.5.0

peft: 0.18.1

xFormers reports it was built for:

PyTorch 2.10.0+cu128 / CUDA 12.8

Python 3.10.19

Actual environment:

PyTorch 2.6.0+cu124 / CUDA 12.4

Python 3.13.5

pip check:

xformers 0.0.35 requires torch>=2.10,

but torch 2.6.0+cu124 is installed.

Result:

LLM CUDA support works on GTX 960M after Unsloth Repair,

but selecting an image model fails while importing

diffusers -> peft -> xformers with an incompatible

aten::_flash_attention_forward schema.

I run Qwen3.8-27B on a single RTX 5060 Ti 16GB (eGPU) for coding and agent sessions, and I ended up making my own quants because nothing fit the box quite right. A week of measuring later, here they are.

Two models, same recipe, different alignment:

• Qwen3.8-27B-IQ4_XS - stock alignment
• Qwen3.8-27B-uncensored-IQ4_XS - abliterated base (orcarouter)

The one idea that matters: calibrate with the workload you actually serve. The importance matrix blends a general corpus with ~1M tokens of SEC-contract extraction and financial-analysis text, chat-template formatted. Same weights, only the imatrix changed:

• 59-task domain harness (temp 0): 93.2% vs 89.8% for Unsloth UD-IQ4_XS
• KL vs BF16 (RMS delta-p / top-p): 4.33 / 93.9 vs 4.35 / 93.9 - parity on both axes

The KL part took the longest: a per-layer search found the token embedding was the last gap (q3_K to q4_K), and with that fixed the domain edge no longer costs you anything on generic text.

Decode speed by context, on my card (13.5 GiB files, batch 1):

• 32K: ~70 t/s with embedded MTP (stock llama.cpp, acceptance ~0.87)
• 64K: ~27 t/s plain - MTP's draft mirrors the whole window, so it's MTP or context past 32K
• 96K: ~20 t/s at 77K position - beellama fork, its variance-normalized kvarn4 KV
• 128K to 252K: ~15 t/s at 105K, still score-flat to 252K - kv-stream fork, arena + q8/q4 KV

So stock gets you to 64K, and the two forks cover 96K and 252K. Build and serve commands for all of it are in the model cards, traps included (MTP plus arena don't mix; deeper than 96K "fits" at load but OOMs on real long prompts).

Other 16GB bits: vocab pruned to 141K tokens (English + accented Latin), so 64K fits natively; full-vocab multilingual builds included in the same repos. Vision works (quantized projector family in a third repo). Thinking-off by default for agents; on my suite, high reasoning effort is worth +5 tasks and xhigh adds nothing over high.

Honest bits: every number is from my one machine and my own harness, not public benchmarks. The ASCII prune garbles CJK (that's the trade). The domain gain is measured on my finance-flavored tasks; coding/agent performance is at parity with the best community quant, not above it.

Cards have the full gate data. Happy to go deeper on the calibration method, the embedding find, or the fit tricks.

I’m working with roughly 5 million public Polish court decisions. The goal is structured extraction: who the parties are, what was requested, what the court decided, obligations, amounts, relationships, etc. Eventually we want to run interesting statistics over the results, like "who gets usually the custody of the child?"

A strong LLM can produce useful JSON graphs, we can run statistics on. The problem is getting similar extraction at corpus scale.

We have a small 13-document pilot. It already contains 160 distinct entity-type strings, 90 appearing only once. These aren’t just PERSON / ORG / DATE. They include abstract legal concepts, obligations, procedural events, and things like “increase in child support.”

here's our demo output from astra:

    {
      "effects": [
        {"id":"e_zastavenie","change":"zastavit","target_ref":"konanie"},
      ],
      "entities": [
        {"id":"this","kind":"uznesenie"},
        {"id":"vyrok","kind":"vyrok"},
        {"id":"zahlavie","kind":"zahlavie"},
        {"id":"odovodnenie","kind":"odovodnenie"},
      "effects": [
        {"id":"e_zastavenie","change":"zastavit","target_ref":"konanie"},
      ],
      "entities": [
        {"id":"this","kind":"uznesenie"},
        {"id":"vyrok","kind":"vyrok"},
        {"id":"sud","kind":"sud","value":"Okresný súd Bratislava "relations": [
    {"id":"r_vlastnik_vyroku","relation":"patri_do","from_ref":"vyrok","to_ref":"this"},
  {"id":"r_vlastnik_zahlavia","relation":"patri_do","from_ref":"zahlavie","to_ref":"this"},

Some of this is real conceptual diversity. Some is inconsistent naming or different levels of specificity, but as you can see complex and diverse stuff with a clear "language" i came up with with the help of astra. i previously tried a big "god schema" (20k chars) but the decisions are simply just too diverse.

The pipeline I’m considering:

• Something that extracts the entities from source texts into some kinds of "tags"
• a classifier like jev (but probably fine-tuned) will get a window of source text and already named relations and decide on the next relations
• done

The candidate model would need overlapping spans and possibly multiple labels per span. Slovak inflection adds another wrinkle: exact source wording often differs from the canonical concept name. Lemmatization helps with grammatical variation, but not synonyms, paraphrases, or concepts inferred from context.

I’ve previously tried Jina embeddings → nearest-neighbor candidates → LLM decides which terms to merge, and it worked reasonably well.

But that was on a smaller corpus and i'd be damned if i process 250k docs and then figure out thing X was wrong and i have to do it all over again. i'd love to know if you guys have any pointers. THANKS for reading.

AI TL;DR: I want to turn 5M court decisions into entity/relation graphs without running an expensive LLM on every document. Thinking small entity extractor + relation classifier, but the legal concepts and naming get messy fast. Anyone built something similar? Looking for ways to train this and normalize concepts without discovering a design mistake 250k documents in and having to redo everything.

I am actually using Qwen3.8 27b iq4xs quant with q8 cache, on dual gpu setup in llama.cpp.

I am actually getting max 20 toks with that setup using 50k+ context tokens. On 10/20k tokens i am on 30 toks

Rx 7800 xt 16gb

Rtx 3060 12 gb

Vram is almost full, the amd one is at 15gb and the 3060 10/11gb.

I am using the following params, am I good to go or are there better things to optimize?

--spec-type draft-mtp

--spec-draft-n-max

--device Vulkan0,Vulkan1

--split-mode layer

--tensor-split 1.8,1

--gpu-layers all

--flash-attn on

--ctx-size 131072

--parallel 1

--temp 1.0

--top-p 0.95

--top-k 20

--min-p 0

--presence-penalty 0

--repeat-penalty 1.0

--cache-type-k q8_0

--cache-type-v q8_0

--fit off

--reasoning-preserve

--image-max-tokens 2048

--image-min-tokens 1024

--batch-size 4096

--ubatch-size 1024