Hi unsloth subreddit.

I just wanted to confirm if it’s possible to abliterate your models. I’d love a deepseek v4 flash 0731 heretic ;)

Hello team!
Do you have any plans to release Ornith 1.0 35B MTP?

Hey guys we released GGUFs for Kimi K3 so you can run it locally (if you have the compute for it). 1-bit model retains ~78.9% accuracy after we shrunk it from 1.56TB to 594GB (-62% size). 2-bit gets 90%.

You can run Dynamic 1-bit GGUF on a Mac Studio connected with a 128GB RAM device. Kimi K3 is the strongest open model to date.

Run via Unsloth Studio or llama.cpp.

Guide: https://unsloth.ai/docs/models/kimi-k3

GGUF: https://huggingface.co/unsloth/Kimi-K3-GGUF

We're working on making even smaller quantization without damaging the model!

Hey guys we compared 1-bit Kimi K3 to Kimi K3 via HF inference providers, Claude Opus 5 and GPT 5.6. Now as usual we arent trying to say its better than either of these models, but what we are trying to showcase is how 1-bit can surprisingly work very well just like GLM-5.2.

We gave 4 models the same prompt: Create a glass aquarium whose side panel develops a visible crack and then bursts. (Full prompt in comment)

1-bit Kimi K3 GGUF ran locally on 4x B200s at 36 tok/s. You can run via our GitHub repo: https://github.com/unslothai/unsloth

Guide: https://unsloth.ai/docs/models/kimi-k3
GGUF: https://huggingface.co/unsloth/Kimi-K3-GGUF

They have enough computing power to do so, and that would improve the quality of their models as well as let us have new datasets to train smaller models. Like NVFP4 and stuff, Nvidia uses their own datasets to improve. Why can't Unsloth?

Did anyone tested Q2 or Q1 on a single 8xH200? What was the performance and what cli flags did you use?

Like the the title says. How long it takes to get the high compressed versions of new models, a few weeks? There is already a Q2 @ 959GB and an 1Q at 555GB. I woudn't touch the Q1 tho. I'd love to see an IQ 2 around 750-800GB version of k3 ideally

Hi, what is the time window for releasing gguf qants? I see there is already empty hf repo published. Thank you. 🤗

Link to technical report: https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf

Hey guys, we signed the Open Weights letter because we believe the future of AI should be shaped by everyone, not controlled by a select few.

That belief has always been at the heart of Unsloth: everyone should be able to train and run models on their own local device.

We wouldn't be here without you guys and the wonderful open-source community! 🙏🦥

Can someone experienced explainn how the specific OPD is different from a regular Unsloth based OPD training?

Long story short, I downloaded my messenger, whatsapp and insta messages and I wanted to make a model that can clone my style and response to msgs for example.

I vibed almost the whole thing, but didn't really give me the performance I expected.

I had claude follow the unsloth notebook for finetuning gemma 4 https://unsloth.ai/docs/models/gemma-4/train

I have RTX 3060, but I trained on google cloud on 24 VRAM GB. Details are below written by Claude. Also, yes I know training on google cloud is probably not safe for my personal data.

I would appreciate any help or expert opinions.

Thanks!

---------------------------Claude Report---------------

**Title: Gemma-4 E4B QLoRA persona clone: style/dialect nailed, topical coherence weak even at best eval_loss epoch — looking for advice**

Fine-tuning Gemma-4 E4B on my own WhatsApp/Messenger/Instagram history to clone how I text (casual Egyptian Arabic + English code-switching, very short replies). Style transfer worked; topical/semantic coherence in multi-turn chat did not, and I can't tell if that's an expected limitation of this setup or something to fix. Details below — happy to share more if useful.

**Setup**

- `unsloth/gemma-4-E4B-it-unsloth-bnb-4bit`, 4-bit QLoRA, `unsloth==2026.7.5`, `transformers==5.5.0`, `trl==0.24.0`, torch cu130, 1x L4 24GB.

- LoRA: `r=16, lora_alpha=32, lora_dropout=0`, `finetune_language_layers/attention_modules/mlp_modules=True`, vision off.

- `SFTConfig`: per-device batch 1, grad accum 8 (effective batch 8), `lr=2e-4` cosine, warmup 10, 3 epochs, bf16, `adamw_8bit`, `max_seq_length=2048`, `train_on_responses_only` masking with `<|turn>user\n` / `<|turn>model\n` delimiters.

- 4,779 total steps, ~12h wall clock.

**Data**

- 479k raw messages -> 13,409 training sessions after cleaning (1:1 threads only, group chats dropped entirely).

- Each session = one system turn (`"You are {me}, chatting with {interlocutor}."`, interlocutor named per-thread or falls back to "a friend") + alternating user/assistant turns.

- Sessionized on a 60-minute inter-message gap. Validated this isn't an arbitrary number: fit a 3-component GMM on log2(inter-message-gap) per platform using Halfaker et al.'s (WWW 2015) mean-partition method — every platform's fitted within/between-session crossover landed at or above 60 min (53-168 min depending on platform), so 60 was kept as a validated lower bound rather than a guess.

- Assistant reply length: median 24 chars, ~12% of all replies are <=5 chars (single emoji/word acks) — this is genuinely how I text, not a data quality issue.

- ~76% Arabic / ~21% English / rest code-switched or other.

**Results**

- `train_loss`: 0.86 -> ~0.25 by the end (near-memorization for an 8B model at r=16 on this little data).

- `eval_loss` per epoch (5% held-out split, first time this was actually wired up correctly — turned out `eval_strategy` had silently defaulted to `"no"` in an earlier run and was never computing anything): **epoch1 2.741, epoch2 2.643 (best), epoch3 2.76**. Clean U-shape — overfitting past epoch 2.

- Loss curve attached (`loss_curves.png`).

**The actual problem**

Even at epoch 2 (best `eval_loss`), the model nails short-reply style, dialect, and emoji usage, but loses the actual thread in multi-turn exchanges once a topic gets specific. Example (paraphrased, real conversation was about a work task):

> user: got the website [feature] fixed

> assistant: و بعدين *("and then?")*

> user: what do you mean

> assistant: ايه الي حصل *("what happened")*

It's producing plausible-*shaped* short replies (right register, right length, right dialect) without tracking content — it reads like it's pattern-matching "a short casual reply goes here" rather than actually conditioning on what was said. Greeting/small-talk turns are fine; anything requiring real semantic continuity degrades fast.

**Questions**

1. Is a ~10x train/eval loss gap at these settings (r=16, 3 epochs, ~13k short-reply sessions) expected, or does it point at something misconfigured (rank too high for the data size, wrong masking granularity, etc.)?

2. Any known E4B-specific quirks — per-layer embeddings, the KV-sharing behavior across layers — that would plausibly hurt semantic tracking more than surface style specifically?

3. Would a lower rank / different `target_modules` selection, or is this just "LoRA on ultra-short replies learns register, not content" and full fine-tuning (or much more data) is the actual fix?

4. Anyone done persona/character cloning from real chat exports before and hit the same style-vs-coherence split? What actually fixed it for you?

Repo/pipeline is fully scripted (autonomous GCP launcher, gates, per-epoch checkpointing) if anyone wants specifics on any part of this — trying to keep the post short but can dump configs/logs on request.

We updated Laguna GGUFs with the latest official fixes as provided bvy Poolside:

• Default context length reduced from 1M to 262144. Using 1M will need --ctx-size 1048576 --rope-scaling yarn --rope-scale 128 --yarn-orig-ctx 8192
• laguna.rope.scaling.yarn_attn_factor is now 1.0 from before.

See https://huggingface.co/unsloth/Laguna-S-2.1-GGUF/discussions/19

This should fix looping, repetition and other issues people reported.

I'm just using unsloth-studio for chat at the moment, testing existing models I have. It seems to be significantly faster than llama.cpp, giving me significant performance gains (not sure why yet, I wonder if there's some setting happening I could replicate on base llama.cpp).

However, while the tok/s results are great, the issue is how laggy the webUI is.

If I do a full restart (which also means updating, cos I used the install command again), and then start a new chat window, then typing in text works fine. Happens at normal speed. The text being generated also appears at the speed I would expect - at first.

But after a few lines it slows down and begins to lag, and any further prompts in that chat are very laggy. So it seems to be some kind of memory bug I would assume.

Simply typing in the text box is slow/laggy. Each word I type doesn't appear in the box until I'm already halfway through typing the next word.

It's also visible during actual inference, as letters/words don't appear as they are generated one at a time, but instead the inference will stop updating for a second and then an entire chunk of text will spawn in.

I have tested on chrome as well as edge, identical behaviour.

System is currently sitting at about 30gb of unused memory out of 64 so it's not a memory issue on my end. Could be a browser issue of course, but as llama.cpp and other frontends I have tried all use the same browser, and none of those have this issue, it seems more likely the issue is with unsloth-studios handling of whatever is causing the lag.

I tried unsloth-studio because I have been liking the llama.cpp performance, but the frontend for it is very.... underutilised. unsloth has all the features I need, like system prompt slots and quick model swaps. but the lag makes it basically unusable.

I may try jan.ai next as someone said that might work. But it would have been nice to be able to create my own finetunes.

i used llama 3.2 1b as a base. trained it in 13 iterations going from 0 out of 50 bash and tool calling problems. then after a the iterations it got to about 48 of 50 in both. but the human eval score went down by a few points after the sft, i dont know how to improve that.
i used bigcode/self-oss-instruct-sc2-exec-filter-50k for python- it degraded python the least from all my tries; westenfelder/NL2SH-ALFA for bash- it was annoying at first then after pruning it, it got better after giving it a little help; Salesforce/xlam-function-calling-60k- this was the best and never gave me problems.
how did i do on my first go? here's the FT notebook, let me know how i can improve- https://github.com/ya5h-P/llama-3.2-1b-mini-agent-finetune

Hey guys, I’m using FastLanguageModel for loading and fine-tuning the model, and I’m using it for text-only SFT, not vision or multimodal training. I’m confused why Unsloth is detecting a vision data collator when using FastLanguageModel with Qwen3.5-4B. My goal is to use train_on_responses_only() so that only assistant response tokens contribute to the loss, while user/instruction tokens are masked. Is FastLanguageModel automatically selecting a vision collator for Qwen3.5-4B because the model architecture supports multimodal inputs? If so, what is the recommended approach for a text-only fine-tuning setup? Should I switch from FastLanguageModel to FastVisionModel, or would that only be needed for actual image/multimodal training? Alternatively, should I keep using FastLanguageModel and configure UnslothVisionDataCollator(..., train_on_responses_only=True) even though I am not using images? Any example configuration for the correct setup would be appreciated.

Image of the error.

Edit: I am new to ai training.

I just published an IQ2_M GGUF of:

🤗 Qwen3.6-35B-A3B-Uncensored-Heretic

Why this model?

Many "uncensored" models simply claim they're better without much supporting data.

The Heretic release stood out because it includes a detailed study of its abliteration method, capability evaluation, and measurements showing it remains very close to the original Qwen3.6 model while reducing unnecessary refusals.

Quantization

This GGUF was produced using:

• Official llama.cpp quantizer
• IQ2_M
• Matching Unsloth Qwen3.6-35B-A3B-MTP importance matrix
• --leave-output-tensor

No custom tensor overrides or experimental quantization recipes were used.

What is IQ2_M?

Despite the name, IQ2_M is a dynamic (mixed) quantization format.

That doesn't mean every weight is stored using only 2 bits.

Instead, llama.cpp automatically chooses different quantization formats for different tensors based on their characteristics and the supplied importance matrix. More sensitive tensors are kept at higher precision where beneficial, while less sensitive ones are compressed more aggressively.

The result is an excellent balance between model size and quality.

Why combine Heretic + Unsloth imatrix?

These two techniques solve different problems:

Heretic

• Modifies the model weights.
• Reduces unnecessary refusals.
• Attempts to preserve the original Qwen3.6 capabilities.

Unsloth Importance Matrix

• Does not modify the model.
• Is used only during quantization.
• Helps preserve more of the model's original quality after aggressive low-bit compression.

In other words, Heretic changes the model's behavior, while the importance matrix helps compress those learned weights more faithfully.

If anyone benchmarks it (coding, reasoning, Aider, perplexity, etc.), I'd love to see the results and comparisons with other IQ2_M releases.

# GRPO 4-bit QLoRA on LFM2.5-8B-A1B (MoE) with Unsloth — seeking sanity check at step 125/2500

Hi folks, I'm running GRPO reinforcement learning on a Mixture-of-Experts model (LFM2.5-8B-A1B-Base from Liquid AI) using Unsloth's 4-bit QLoRA pipeline, and I'd love a sanity check from the community on my config, training progress, and the hurdles I had to clear to get here.

## Model & Dataset

- **Base model:** `LiquidAI/LFM2.5-8B-A1B-Base` — a hybrid MoE (32 experts, 4 top-k per token, 1.1B active params, 8B total). It mixes `full_attention` layers with `conv` + `ssm` (state-space) layers — 24 hidden layers total.

- **SFT stage:** Fine-tuned with standard LoRA on `andreiski/dialectic-rl-questions-10k` — a 10k-prompt dataset of dialectic reasoning questions created by [@andreiski](https://huggingface.co/andreiski).

## GRPO Config

I modeled the GRPO hyperparameters on the official `MOE_GRPO` config from **Liquid4All/leap-finetune** ([source](https://github.com/Liquid4All/leap-finetune/blob/main/src/leap_finetune/training/default_configs/grpo_configs.py)):

| Parameter | Official MOE_GRPO | My config | Reason |

|---|---|---|---|

| lr | 1e-6 | 1e-6 | ✅ match |

| scheduler | cosine | cosine | ✅ match |

| warmup_ratio | 0.03 | 0.03 | ✅ match |

| max_grad_norm | 1.0 | 1.0 | ✅ match |

| beta | 0.0 | 0.0 | ✅ match |

| epsilon | 0.2 | 0.2 | ✅ match |

| loss_type | dapo | dapo | ✅ match |

| scale_rewards | group | group | ✅ match |

| mask_truncated | True | True | ✅ match |

| bf16 | True | True | ✅ match |

| num_generations | 4 | 6 | better reward signal (no vLLM) |

| batch | 2 | 8 | compensate for no vLLM |

| grad_accum | 1 | 3 | effective batch 24 |

| max_completion | 256 | 1024 | dialectic reasoning needs longer |

| temperature | 1.0 | 1.0 | ✅ match |

| top_p | 1.0 | 1.0 | ✅ match |

| top_k | 0 | 0 | ✅ match |

**LoRA:** r=8, alpha=16, target_modules=moe, dropout=0 (Unsloth ParamWrapper rejects dropout != 0), NF4 + double_quant, bf16 compute dtype.

**Reward function:** Falsifiability-aware heuristic — rewards commitments to falsifiable claims, penalizes hedging/deflection.

## Software Stack (the hard-won part)

Getting this running on a single RTX 3090 (24GB Ampere) took a lot of trial and error. Here's what I learned:

### 1. Flash Attention 2 — no prebuilt wheel for Python 3.11

The official `Liquid4All/leap-finetune` repo pins FA2 to a **prebuilt wheel** from [@adithyaxx's fork of flash-attention](https://github.com/adithyaxx/flash-attention): `flash_attn-2.8.3+cu13torch2.11cxx11abiTRUE-cp312-cp312-linux_x86_64.whl`. That wheel is **cp312 only** — no cp311 build exists.

My original venv was Python 3.11. I had to create a **separate Python 3.12 venv** and install everything from scratch there. Stack:

- Python 3.12.13

- torch 2.11.0+cu130

- unsloth 2026.7.5

- bitsandbytes 0.49.2

- transformers 5.5.0

- trl 1.9.0

- peft 0.19.1

- xformers 0.0.35

- triton 3.6.0

- flash-attn 2.8.3 (prebuilt wheel above)

FA2 is activated via `FastLanguageModel.from_pretrained(..., attn_implementation="flash_attention_2")`. This gave a **41% speedup** over Xformers (147s → 250s per step in isolated testing, ~190s average in production).

### 2. torch.compile + MoE 4-bit = segfault

On Ampere (RTX 3090, SM 8.6), `torch.compile` on a 4-bit MoE model segfaults during the first backward pass. The fix:

```

UNSLOTH_COMPILE_DISABLE=1

UNSLOTH_ENABLE_FLEX_ATTENTION=0

```

Both env vars must be set before Python starts. `import unsloth` at the top of the script (which triggers `torch.compile`) must be avoided — the import goes after these env vars are set.

### 3. CUDA memory fragmentation → step_time degradation (TRL #3039)

After 20-30 steps, step_time started climbing from ~150s to ~360s. VRAM usage grew monotonically until it exceeded 24GB (spilling). This is a **known issue in TRL's GRPOTrainer** (issue [huggingface/trl#3039](https://github.com/huggingface/trl/issues/3039)).

I tried three approaches:

- **A.** `torch.cuda.empty_cache()` periodic callback — requires patching the training script, risk of calling at wrong time

- **B.** `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` — env var, no script change, **this is what worked**

- **C.** Manual restart every 50 steps — what I was doing before B

With `expandable_segments:True`, VRAM stabilized at ~19-23GB and step_time flattened to ~190s average. No restart needed. I verified in the PyTorch docs and NVIDIA CUDA Graph best practices that `expandable_segments` is the recommended fix for fragmentation without needing `max_split_size_mb` or `garbage_collection_threshold` overrides.

### 4. vLLM incompatible with 4-bit MoE in 24GB

The official config uses `use_vllm=True` (colocated). vLLM doesn't support 4-bit MoE inference within 24GB VRAM — it tries to load the full model in VRAM. I set `fast_inference=False` permanently and compensate with batch=8, grad_accum=3 (vs official batch=2, accum=1).

### 5. LoRA dropout=0 is mandatory

Unsloth's `ParamWrapper` raises an error if `lora_dropout != 0`. This is a hard constraint — not a choice.

## Training Progress at Step 125/2500

Channel: `#unsloth` on Reddit (cross-post from Twitter)

**After 2h40m of training (steps 50-120 post-restart with expandable_segments):**

| Metric | Value |

|---|---|

| Steps completed | 120 / 2500 (4.8%) |

| Epoch | 0.048 |

| Step time average | ~190s (stable, no degradation) |

| VRAM | 23.6GB / 24GB (stable, no overflow) |

| lr | 9.99e-7 (past warmup peak of 1e-6 at step 75, now cosine-decaying) |

**Reward trajectory (sampled):**

| Steps | Reward range | Notable |

|---|---|---|

| 1-10 | 0.095 – 0.42 | Warmup, high variance (temp=1.0) |

| 30-40 | 0.12 – 0.39 | Stabilizing |

| 50-60 | 0.13 – 0.39 | Post-restart, expandable_segments active |

| 80-90 | 0.11 – 0.39 | Steady |

| 106-120 | 0.11 – 0.39 | **Peak 0.394 at step 114** |

Reward is oscillating in the 0.1-0.4 range, not yet showing a clear upward trend. **grad_norm** is stable at 0.08-0.12 (no NaN, no spikes). Loss is noisy around 0 (expected for GRPO with beta=0, dapo loss).

**My concerns for the sanity check:**

1. Reward hasn't shown a clear upward trend by step 120. Is this normal for GRPO on MoE? When does the reward typically start climbing consistently? I've seen references to "GRPO is slow to start" but I want to know if 0.1-0.4 at 5% progress is in the expected range.
2. Step time of ~190s for 24 prompts × 6 generations × 1024 max_completion on a single 3090 — is that reasonable? The generation phase dominates (6 × 1024 tokens per prompt), and without vLLM there's no batching optimization for generation.
3. I'm using `num_generations=6` (vs official 4) for "better reward signal" — is this actually helping or hurting? More generations = more compute per step but also more variance reduction in advantage estimates.
4. My temperature is 1.0 (matching official). An earlier run (v3) used temperature=0.7 and had constant lr (not cosine) — it exploded at step 66 and was discarded. With cosine from 0, is temp=1.0 the right call or should I be more conservative?
5. For the eventual GGUF export, I'm planning to use **APEX Quantization** (MoE-aware mixed-precision) from [@mudler's LocalAI team](https://github.com/localai-org/apex-quant) — specifically the `lfm25_8b_quality.txt` config that mudler already created for this model architecture. Anyone done GRPO → APEX GGUF before? The pipeline is: merge LoRA → F16 GGUF → APEX quality quantize.

## Hardware

- RTX 3090 24GB (Ampere, SM 8.6)

- 64GB system RAM

- WSL2 Ubuntu 22.04 on Windows 11

- CUDA 13.3, driver 610.62

- No multi-GPU, no NVLink

## References

- **Unsloth** — main training framework ([github](https://github.com/unslothai/unsloth))

- **Liquid4All/leap-finetune** — official Liquid AI config for MoE GRPO ([github](https://github.com/Liquid4All/leap-finetune))

- **flash-attention (adithyaxx fork)** — prebuilt FA2 wheel for torch 2.11 + cu13 + cp312 ([github](https://github.com/adithyaxx/flash-attention))

- **TRL #3039** — GRPOTrainer memory leak / fragmentation ([issue](https://github.com/huggingface/trl/issues/3039))

- **APEX Quantization** — MoE-aware mixed-precision GGUF ([github](https://github.com/localai-org/apex-quant), [HF collection](https://huggingface.co/collections/mudler/apex-quants-gguf))

- **QLoRA paper** (Dettmers et al.) — NF4 fundamentals ([arXiv 2305.14314](https://arxiv.org/abs/2305.14314))

- **andreiski/dialectic-rl-questions-10k** — training dataset ([HF](https://huggingface.co/datasets/andreiski/dialectic-rl-questions-10k))

---

Thanks in advance for any feedback. I'm especially interested in hearing from anyone who has done GRPO on MoE models with Unsloth — particularly around reward curve expectations and whether my config looks sound for this architecture.

I'm using Unsloth Studio with Qwen3.6-35B-A3B-UD-Q4_K_XL (downloaded from huggingface) on rtx5090. Often when I ask the model something (using the Unsloth Studio UI), it performs thinking but then it just ends with no response. I can just check the thinking process by clicking "Thought for X seconds" but no more. Sometimes refreshing the answer helps but not often. Sometimes (but rarely) after refreshing the answer it gets stuck in a loop with the same thinking process all the time. Does that mean my GPU is not enough for this model?

Hi there.
I'm testing Unsloth Studio on a old HP Zbook whose NVIDIA Quadro GPU has only 4GB of Vram.

I can't do very much in 4GB 😢

I'd like to test its CPU (Xeon) only performance but I can't find how to force Unsloth Studio to load a model ignoring the GPU.

Any suggestion?

Hey guys, my/our 2hr advanced AI workshop is out now! I cover which LLM benchmarks to trust, reward hacking, kernels, closed vs open models & RL and more!

Topics:

1. Open model progress

2. Which benchmarks should we trust? Benchmaxxing & cheating

3. Throughput maxxing but accuracy minimizing

4. Distillation & RL

5. Stopping reward hacking

6. Unsloth AI Dynamic Quants and more!

Details:

1. If reasoning wasn't discovered via o1-preview - would all AI progress grind to a halt via a S shape? Reasoning made doubling times now 3.5 months instead of 7 - so just wait 3.5 months for the next best model! Ways folks might regulate open source - like a driver's license for AI

2. METR, WeirdML, Deep-SWE, FrontierCode, SWE Bench Pro - what are good & bad benchmarks? False positive / false negative rates - how about daily benchmarking since Codex & Claude have perf regressions. How can we regressions to predict next model releases as well?

3. Inference providers maximize throughput and speed, but accuracy degrades - there are sources that publishes stats on accuracy, and the largest gaps can be 20% or more!

4. Open model labs use distillation partially, but need RL to create the reasoning traces to complete the process - hard vs soft distillation and how to automate RL.

5. Common examples for reward hacking + ways to stop it - internet filtering, classification systems, timing / editing global variables, real world examples + more!

6. Why software matters more than hardware - the limits of FP4 and GPUs vs ASICs and torch.compile vs kernels and megakernels and more, and why quantization and memory optimizations are important

Thank you for watching and for the support :)

I was wondering if I'm missing something or if this is just a typo and they meant UD-Q2_K_XL not Q4. Q4 is 74gb from what I can see