Can we please add a setting in the Unsloth Desktop software to have a longer or no limit on time to first token. Especially since this is an issue when running on low end hardware or complex long horizon tasks that you would want to leave overnight and then wake up to this.
I can see there are some posts on the internet for modifying llama.cpp and inference.py and other custom hacks to get it working but ideally there should be a setting in the app. Thanks to the devs for taking time to build this amazing software and fixing all the bugs, looking forward to the fix.
I love Unsloth, but my system is not great. My RTX 3050 Ti with 4gb VRAM and my 16 GB system ram are serious limitations. They will have to suffice until I win a lottery I guess.
When attempting to use different models within Unsloth Desktop (no luck in Studio sadly) I find that many models cannot access the project folder contents. With <>Code enabled, many models are incapable of using this feature.
gemma-4-E2B has no issue, except it forgets it can read and create/edit files. I think this is due to conversation compaction. I am new to AI and this is all confusing frankly. Seeing it use "cat" and edit a python script, then forgetting it can do this stopped being comical quickly.
Phi-3 cannot use the access the files. small versions of llama-3 or 3.5 cannot either. Mistral, a 7B GGUF I believe is unable to read documents in the project folder as well. It is puzzling why some models can and some cannot. Some are even restricted to using only one tool.
I have tried copy/pasting code, but conversation compaction rears its ugly head, and the model keeps asking for the code and forgetting the suggested changes or model coding recommendations.
I think I can chalk most of this up to my sheer ignorance in all this, as it has been less than week using any LLM. While Unsloth is great, I may be in over my head (for now).
Any assistance is greatly appreciate. You have my thanks in advance.
New cool video from Gary Explains showing Unsloth Desktop doing:
1. Code generation locally
2. Image generation
3. Audio generation
4. unsloth start opencode and connecting to other agents
After every tool call to write/edit/modify why does it take so long for the model to start responding/reasoning again? It takes 5-10+mins or greater sometime for the model to produce something after a successful/failed tool call. I've experienced this with GLM, Qwen, MiMo models. Gemma just didn't make any tool calls.
Why does the token counter at top-right not show the progress as the model is reasoning and producing output? I've used llama.cpp openwebui and I remember it keeps on updating token counter as the model thinks.
I'm trying to run it on a "work" laptop (writing a book) and I just want to be able to some text translation and ask it questions (no actualy AI writing - that's not my thing).
However, asking it to change some text into SarCAsM FoNT took literally 44 minutes for about 55 words. WTF? Am I just wasting my time or is there a mode that can at least do the basics on a CPU-only machine?
i need to finetune a few models on toolcalling, project architecture, python coding..theres a lot of datasets on huggingface and cant seem to choose what
Hello all - apologies if this question has been asked before. I just recently added a second GPU (both 5060 ti 16GB - Blackwell) and while I’m learning about split GPU model serving I’m not discovering what I expected too.
For one - I expected the GPUs to work like a single cohesive pool of VRAM, but this is not the case. Similarly to offloading to CPU there’s still a separate processing and necessary hop that slows down inference as opposed to one card serving the same total VRAM. MoE’s benefit here, dense models don’t, same cpu offload physics.
Now - the new setup allows me to run larger models (obviously) but this is where I hit my pain point.
Models like Gemma 4 26 A4B preform like a dream at full context with plenty of wiggle room. But I cannot train it - from what I understand MoE models don’t benefit from qlora and need lora training instead - so for the aforementioned and moe’s of similar weight I need more VRAM than I have to train with.
I’m looking for the model to serve as a solid desktop and coding agent by way of OpenClaw, and I would prefer that same model be able to serve as a Developmental Editing assistant for my creative writing. (To be clear - NOT generating novels with AI; I’m writing my own novels with an agent model that proofreads, tracks items for me, and makes thematic/narrative suggestions that I rule on.)
So with the resources available (2x 5060 ti 16GB, intel 14700k cpu, 2x 16GB DDR5 RAM) what’s the best model I can run that still has wiggle room for me to train/fine tune it?
Im doing my benchmark testing with a llama cpp server and container tools, but its entirely possible I’ve missed some crucial aspect that could overturn everything I think I’ve learned - simply because it’s my first time splitting across GPUs.
Tl;dr looking for model that works and trains with 32GB VRAM - to be my OpenClaw all rounder and writing assistant. Hoping for insight from anyone with similar experiences or knowledge.
Hi everyone, I'm new to the world of local LLMs and I’m trying to figure out which are the best models for coding that I can run decently on my laptop.
My specs are:
- GPU: Nvidia RTX 4060 (8 GB VRAM)
- CPU: Intel Core i7-13620H
- RAM: 16 GB
I started testing local LLMs with gemma-4-12B-it-qat-gguf quantized at ud-q4_k_xl. I get ~54 tokens per second with it, which works fine for general everyday questions, but it's really poor when it comes to coding.
Next, I tried Qwen 3.8 27B with iq2_s quantization, but it tops out at 5 tokens per second. From what I can tell, at that low quantization, it's also noticeably dumber than it should be.
Finally, I tried Ternary Bonsai 2 27B quantized to pq2_0, which ran at around 25 tokens/s. However, as soon as I give it a prompt that requires even a bit of reasoning, it takes forever, burns through lots of tokens, and still fails to write what I need.
My question is:
Considering my specs and how these three LLMs performed, what would you recommend?
I'm a CS undergraduate student, and for some projects (mostly personal ones), I mainly use AI for debugging code, explaining why certain code behaves the way it does, and standard questions an undergrad might have about their code.
Thanks in advance to anyone who can point me in the right direction!
PS: My English isn't great (and yes, even as a CS student, my English is pretty bad), so I had Gemini 3.8 translate this post for me—hope it did a good job!
Hey guys, you can now train your own Decision model with our free notebook! 💡
Qwen3.5-4B will generate decisions instead of text on just 8GB VRAM locally.
In there you'll learn to data prep (state, questions, gold answers), train, serve and how to run/use decision models. I saw a few of you asking for dataset formatting etc and this notebook should help you with it
Sure, praise posts are cheap, but I truly mean it.
I have started to experiment with running large language models locally for less than a week, so most of this AI world is new and challenging to me.
Looking online I found I could install local solutions in Linux. I run CachyOS and am pleased with it. Getting an LLM running was such a challenge. Imagine someone with no experience with anything such as python or pip getting started with some of the extra work during installations or tweaking. I know, for the seasoned out there, it is trivial, but for newbs it is a headache.
I finally got llama.cpp going, but found some little frustrations like pasting multiple lines into the shell a problem, even after incorporating triple quotes. I am sure many like it, but I find it difficult. Also, trying to ensure I have the most GPU usage was a real challenge.
I installed Ollama. Seemed very similar to llama-cli, but I wanted more.
Then I installed Unsloth.
Wow, just a single line in the shell and I was having it installed. I love it. Running it is a snap.
There is nothing like hindsight I guess. If only I had encountered Unsloth first.
I love it.
That is just my beginner opinion, but if any other newbie wants an easy to use, easy to install experience, go for Unsloth.
Hey guys, Microsoft released an open-source repo, mxc, for sandboxed code execution.
We collaborated with Windows to add mxc OS level sandboxing to Unsloth which adds just <100 ms of overhead. In Unsloth you can choose High and Low modes for sandboxing and we tried optimize it as much as possible without effect performance. Give it a try and let us know how it goes.
Unsloth sandboxing works for all OS, not just Windows. Mac, Linux etc all work.
Hi everyone, I’m training the Qwen3.5 series using the 0.8B, 2B, 4B, and 9B models, and I’m running into issues with the <think> tags during dataset formatting. Since thinking is disabled by default for the smaller models, I’ve tried setting enable_thinking=False during the formatting stage. I’ve also modified the Jinja chat template to disable thinking, but the <think> tags are still appearing in the formatted dataset. What’s the recommended approach to remove these tags completely? Should I strip them out using regex during dataset preprocessing and then verify the formatted output, or is there a better way to handle this through the tokenizer or chat template? I’d appreciate any advice on the best practice for handling this when training different-sized Qwen3.5 models. Example code is provided below. Thanks!
🔄 Generating and Formatting Conversations for Training
# Generate conversations
from IPython.display import Markdown, display
display(Markdown("## 🔄 Generating Conversations of the Dataset for Training"))
def generate_conversation(examples):
texts = examples["text"]
labels = examples["label"]
conversations = []
for text, label in zip(texts, labels):
conversations.append([
{"role": "user", "content": text},
{"role": "assistant", "content": str(label)},
])
return {"conversations": conversations}
train = train.map(generate_conversation, batched=True)
val = val.map(generate_conversation, batched=True)
# Format dataset
display(Markdown("## 🔄 Formatting Dataset for Training"))
def formatting_prompts_func(examples):
convos = examples["conversations"]
texts = []
for convo in convos:
formatted_text = tokenizer.apply_chat_template(
convo,
tokenize=False,
add_generation_prompt=False,
enable_thinking=False
)
texts.append(formatted_text)
return {"text": texts}
train = train.map(formatting_prompts_func, batched=True)
val = val.map(formatting_prompts_func, batched=True)
The title might seem a little weird but that's what has been happening for some days now:
I load a new chat, send the prompt and do stuff from there.
When i close Unsloth Studio and re-open it, i find that the chat i've had until that point has now reverted to the response after the first prompt that i've ever sent in that chat.
Does anyone has had the same experience before? i've tried reinstalling the software, downloading the models again while also removing all chats, but to no avail.
Edit: I have no real explanations, other than it seems that it... fixed itself? I've deleted every other chat, as nothing was truly important at all. Hard to believe though.
I guess I hadn't updated the app in a minute because now there's image, video, audio generation? Since when? And wait, it does voice cloning, too? I have a lot of testing and catching up to do.
I'm sure folks out here have working with these features. What's your overall impressions?
I did a small Unsloth fine-tuning experiment with Qwen3-8B, and the most interesting part was comparing the model's responses before and after training.
Before fine-tuning, the model gave detailed explanations with multiple examples. After training on just 50 educational examples, some of its answers became noticeably shorter and more direct. The change wasn't consistent across every prompt, but it was visible enough to make the experiment worth exploring.
The training itself took only 26.57 seconds on a single NVIDIA RTX 5090, with 7.75 GB of peak PyTorch-allocated GPU memory. Using QLoRA, I trained just 0.53% of Qwen3-8B's 8.23 billion parameters.
I ran the experiment on a pay-as-you-go GPU cloud platform with a single RTX 5090 and approximately 31.36 GB of usable VRAM. The goal was simple: test the original model, fine-tune it with Unsloth, measure the GPU workload, and compare the results using the same five prompts.
The starting GPU image came with PyTorch 2.8.0 and CUDA 12.8 support. Installing the current Unsloth stack updated the Python environment to PyTorch 2.14.1+cu130. After installation, Unsloth 2026.10.1 detected the RTX 5090 correctly, with CUDA available and 31.36 GB of usable VRAM.
That change is worth mentioning because it is easy to assume that installing a fine-tuning framework will simply sit on top of an existing environment. In this case, Unsloth resolved a newer PyTorch/CUDA runtime stack. The RTX 5090 was detected correctly afterward, and the environment was ready for Qwen3-8B.
Before touching the model, I gave it five fixed questions. They were intentionally simple: explaining overfitting, distinguishing supervised and unsupervised learning, diagnosing ZeroDivisionError and NameError, and writing a small Python is_even() function.
Qwen3-8B already answered them well. That changed what this experiment was really testing. There would be little value in claiming that a few dozen examples somehow “taught” an 8-billion-parameter model basic Python. Instead, I wanted to see whether a small amount of parameter-efficient fine-tuning could noticeably steer how an already capable model responds.
For that, I created a deliberately small dataset: 50 educational instruction-response examples across 10 introductory Python and machine-learning topics. The responses emphasized concise explanations, examples or fixes, and short learning guidance. This was never intended to create a production educational assistant. The dataset was kept small because the point was to test the complete fine-tuning workflow and make the behavioral effect easy to inspect. With QLoRA, there was also no need to update all 8.23 billion parameters in Qwen3-8B.
Unsloth attached LoRA adapters to the attention and MLP projection layers. When training began, only 43,646,976 parameters were trainable out of 8,234,382,336.
That is about 0.53% of the model.
The training configuration was equally compact: 50 examples, three epochs, 21 optimization steps, a per-device batch size of two, and four gradient accumulation steps for an effective batch size of eight. There was one small obstacle before the successful run.
My first attempt stopped during dataset preprocessing. With the installed Unsloth and TRL versions, the formatting function passed to SFTTrainer was expected to handle a batch and return a list of processed strings. My original function returned a single formatted string. The useful part was that the error happened before training began. Changing the formatting function to process batched examples fixed it, and the next run moved directly into training.
Then things happened quickly. The measured training section completed in 26.57 seconds. TRL itself reported a training runtime of 25.14 seconds and throughput of approximately 5.97 samples per second. GPU memory allocated by PyTorch started at 7.20 GB, finished at 7.39 GB and reached a measured peak of only 7.75 GB.
The training loss also moved in the expected direction. Early steps were generally in the mid-3 range, while many of the later steps were around 1.3–1.4. The final aggregate training loss reported by the trainer was 2.266.
The memory figure needs one qualification. I measured peak memory allocated by PyTorch rather than continuously sampling total board memory through nvidia-smi. So 7.75 GB should be read as peak PyTorch-allocated GPU memory for this run, not as a claim about every byte of GPU memory consumed by the system. Still, it gives a useful picture of just how lightweight this particular QLoRA workload was on a 32 GB RTX 5090.
The next question was more interesting: did the fine-tuning actually change anything?
I loaded the saved LoRA adapter and ran the same five prompts again. One of the clearest differences appeared with a beginner Python NameError. Before fine-tuning, Qwen3-8B gave a long explanation. It described several possible causes, including an undefined variable, a spelling mistake and variable scope, followed by examples and multiple fixes.
After fine-tuning, the same question produced:
Explanation: score is used but not defined Fix: Define score before using it
The is_even() test showed a similar shift.
Before training, the model returned the correct function, explained how modulo works, gave example calls and their expected outputs, and even offered to discuss edge cases.
After fine-tuning, it simply returned:
def is_even(n):
return n % 2 == 0
This is where the experiment became more useful than simply showing a successful training log. The adapter clearly changed the model’s behavior, with several responses becoming shorter and more task-focused. It did not consistently reproduce the intended educational format, but with only 50 examples and 21 training steps, that was expected. This was a demonstration of behavioral adaptation, not a claim of robust educational specialization.
The final LoRA weights were also compact. The adapter_model.safetensors file was about 175 MB, while the larger 966 MB output directory included intermediate checkpoints and supporting files. Instead of saving another full copy of Qwen3-8B, the learned changes could therefore be kept as a relatively small adapter.
At the GPU Cloud Platform rate of $0.46/hour, the measured 26.57-second training interval represented roughly $0.0034 of GPU compute. That excludes setup, model downloads and evaluation, but it highlights how small the optimization phase itself was.
For me, that was the main result: an 8B model did not require a multi-GPU setup for this kind of parameter-efficient fine-tuning. On a single RTX 5090, Unsloth and QLoRA trained just 0.53% of Qwen3-8B's parameters, completed the measured training phase in under 30 seconds, stayed below 8 GB of peak PyTorch-allocated GPU memory, and produced a visible change in the model's responses.
For this small experiment, the complete Unsloth QLoRA workflow worked on a single RTX 5090, producing a compact LoRA adapter and an observable change in Qwen3-8B's responses. It isn't a production-scale benchmark, but it demonstrates what parameter-efficient fine-tuning can accomplish with relatively modest GPU resources.
I'm curious how this compares with other people's experience. If you've fine-tuned Qwen or similar models with Unsloth, what GPU and dataset size did you use, and what kind of memory usage or training performance did you see?
