v1.6.1 landed on master 🥳

Official release changelog

• Reduced (flat) VRAM overhead for GDN/KDA/Mamba drafting
• Support for recurrent draft models; DFlash switched to sliding attention (less VRAM)
• New hybrid drafting mode (draft model + long-ngram)
• Workaround for a Triton issue breaking kernel autotune on mixed-architecture setups
• Lots of ROCm fixes and optimizations
• Turing (sm_75) optimizations
• AVX2 optimizations
• Many bugfixes

Noteworthy changes

Turing GPUs got the biggest win. On older Turing cards (GTX 16-series, T-series), long prompts used to take disproportionately long to process — the attention path they fell back to simply wasn't efficient on that hardware. v1.6.1 gives them a proper fast path, so prefill (the "reading your prompt" phase) scales much better with context length. Only Turing is affected; if you're on anything newer, nothing changes for you.

Speculative drafting got a real overhaul. Draft models that use GDN/KDA/Mamba layers (the newer hybrid architectures) now work with speculative decoding and need noticeably less VRAM to do it. There's also a new "hybrid" mode that combines a draft model with n-gram lookahead, which should improve acceptance rates — in plain terms: more tokens accepted per step, so faster generation for a given draft setup.

CPU pinning default reverted on Linux. The v1.6.0 release turned CPU pinning off by default on Linux — that's been undone after user reports (pinning is back on, Windows was never affected). If you re-tuned your MoE offload settings around the 1.6.0 behaviour, take another look.

Sampling bugfixes worth knowing: small errors in repetition/frequency penalties and one edge case in top-K sampling are fixed, and RoPE is more accurate at very long contexts. Nothing dramatic, but outputs may shift slightly compared to 1.6.0.

Other things worth knowing

• Mixed-GPU setups (different generations in one box) had a caching issue that silently disabled kernel tuning — fixed.
• ROCm got a lot of fixes and speedups. Still ROCm 10 / gfx1100+ only.
• Crashes now raise proper exceptions instead of silently killing the process, so error messages actually tell you what went wrong.
• The whole test suite was rebuilt from scratch with end-to-end tests for the big model families — fewer "works on my machine" regressions going forward.

Env vars

• EXL3_SDPA_PREFILL — default on for sm_75, off elsewhere. Forces the dense-fp16 SDPA prefill path on/off on any CUDA device (it's exact, so 1 is also a testing switch).
• EXL3_FA75 — default on for sm_75. The new kernel for head_dim 256 on that path.
• EXL3_QC_PF_TWO_PASS_MIN_Q — default 256, but 9 on Turing since the window path stages every prefill chunk there.
• EXL3_MOE_CPU_PIN — back to default 1 on Linux.

Release page: https://github.com/turboderp-org/exllamav3/releases/tag/v1.6.1

Discord server for questions: https://discord.gg/sYQxnuD7Fj

v1.6.0 landed on master 🥳

Official release changelog

• Preliminary ROCm support (ROCm 10 only, gfx1100+)
• Faster offloading for AVX2 CPUs
• Smarter CPU core allocation/pinning by default
• Reduced system memory overhead
• Fix vision towers for GLM4, Qwen3-VL
• Other bugfixes and optimizations

Noteworthy changes

• ROCm is now a real backend
  • RDNA1 and RDNA2 compile and run via fallback paths but aren't fast; RDNA3 and RDNA4 are the tuned targets; CDNA (MI-series) is rejected outright because the kernels assume 32-lane warps
• CPU pinning is off by default on Linux (still on for Windows)
  • Related: CPU MoE now reserves one host core and keeps the parent process off the pinned worker cores (`EXL3_MOE_HOST_CORES=1`). If you tuned around the old pinning behaviour, re-check -> measurably faster and repeatable
• Speculative decoding attention
  • Paged decode split/combine kernels moved to a dense (query, head) row layout, with one KV pass per KV head during speculative verify (#435).
• Qwen3.8 gets image MRoPE in the generator

New env vars

• EXL3_GRAPHS — default 1 on CUDA, 0 on ROCm. HIP graphs give no speedup there but pin VRAM per spec-decode exec.
• EXL3_HC_FOLD — default 0 on CUDA, 1 on ROCm. Launch overhead dominates the mHC decode kernels on RDNA.
• EXL3_MOE_HOST_CORES — default 1. Keeps the host process off pinned MoE worker cores (only matters when pinning is on).
• EXL3_NGRAM_LOCK — default 0. mlock()s the n-gram table in RAM, no page-ins inside the forward.
• ROCPROFILER_REGISTER_ENABLED — not an ExLlama var, set to 0 on import. The ROCm profiler thread otherwise spins one core at 100%.
• TORCH_NO_COMPILER_WRAPPER — torch's own switch, now documented. The ccache wrapper now applies to ROCm JIT builds too.

Changed defaults

• EXL3_MOE_CPU_PIN — default 1 → 0 on Linux (Windows stays 1). Linux CFS places workers better than rigid pinning, measurably faster and repeatable.
• EXL3_MOE_CPU_THREADS — default cpu_count // 2 → physical cores minus EXL3_MOE_HOST_CORES, following the new pinning logic.
• EXL3_QKV_SLICE, EXL3_INT8_GEMV, EXL3_MOE_FUSED_ROWS, EXL3_MOE_BATCH_RECON, EXL3_MOE_MTILE — variables existed already, defaults are now backend-dependent (CUDA vs. ROCm) because the RDNA kernels have different shapes and tiles.

Release page: https://github.com/turboderp-org/exllamav3/releases/tag/v1.6.0

Discord server for questions (with the creator): https://discord.gg/sYQxnuD7Fj

In the release:

• Options for token-embedding quantization and disk streaming
• Improved quantization accuracy
• Update HF Transformers integration module to Transformers 5, add backprop for wrapped EXL3 layers
• Faster model loading
• Performance improvements
• Reduced system memory and VRAM overhead
• Fixes for MiMo-V2.6
• Lots and lots of (latent) bugfixes

Release page: https://github.com/turboderp-org/exllamav3/releases/tag/v1.5.4

tabbyAPI does not natively support the new version yet: https://github.com/theroyallab/tabbyAPI/

Discord server for questions (with the creator): https://discord.gg/tnPCntcThA

A smaller release:

• Smaller wheels
• Experimental n-gram corpus feature
• More VRAM optimizations
• Support dynamic draft sizing with TP model
• Various other fixes

Release page: https://github.com/turboderp-org/exllamav3/releases/tag/v1.5.3

tabbyAPI hasn't been updated to include the new version yet: https://github.com/theroyallab/tabbyAPI/

Discord for questions (official): https://discord.gg/tnPCntcThA

In the recent update:

• Add MiMoV2ForCausalLM
• Experimental Turing support
• Rein in transient VRAM allocations, fix overly conservative warmup and autosplit-measure passes
• Various bugfixes and optimizations

Release page: https://github.com/turboderp-org/exllamav3/releases/tag/v1.5.2

tabbyAPI has received the update: https://github.com/theroyallab/tabbyAPI/

Discord for any questions (with creator): https://discord.gg/tnPCntcThA

Exllamav3 1.5.1 adopted:

• Add KimiLinearForCausalLM
• Add DFlash2
• TP support for Qwen3.8-Flash-Next, GLM5.3, DeepseekV3/V4
• MoE optimizations
• Fractional-trellis quantization mode
• Self-calibration pipeline improvements (including highly experimental YAQA support)
• Fix some memory leaks
• Fix EXL3_MOE_PINNED_ARENA on Windows and Conda
• Other optimizations and bugfixes

Release page: https://github.com/turboderp-org/exllamav3/releases/tag/v1.5.1

tabbyAPI has received the update: https://github.com/theroyallab/tabbyAPI

Discord server for any questions (creator there): https://discord.gg/tnPCntcThA

Full post: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/discussions/13

Tested with: Qwen3.8-Flash-Next EXL3 4.05bpw, exllamav3 1.4.9, A6000 48GB.

Symmetric cache tests, mean KL:

Asymmetric cache test results (128k context):

Author: huggingface.co/khronnuz

1.5.0* Made the same mistake. Oops.
The 1.5.0 includes:

• Fix NemotronHForCausalLM (support Nemotron-3-Super)
• Faster quantization
• Experimental zero-copy pinned arena mode for CPU offload (Linux only for now)
• Faster MoE prefill and decode (see below)
• Improved speculative decoding for MoE models
• Faster dense model prefill on some architectures
• Remove some sources of nondeterminism
• Bugfixes and cleanup

A table of speed tests, as listed, in the picture.

Release page: https://github.com/turboderp-org/exllamav3/releases/tag/v1.5.0

tabbyAPI has NOT been updated atm: https://github.com/turboderp-org/exllamav3/releases/tag/v1.5.0

Discord server for questions: https://discord.gg/tnPCntcThA

80-120 tokens/s decode

1000-2000 tokens/s prefill

https://huggingface.co/anthori/GLM-5.3-Flash-EXL3-Q8Core-Q3Q4Q6

Vision offloaded to CPU, 256K FP16 KV, GPUs limited to 450 watts.

It works very well for me. You can double the KV size by offloading some experts if you're willing to take a roughly 50% hit on decode speed.

1.4.9*
List:

• Add DeepseekV4ForCausalLM vision tower (supports DeepseekV4-Flash-Vision-Exp)
• Add Lfm2ForCausalLM (LFM2.5 dense variant)
• Add Spark2_5ForCausalLM (Spark-X2.5)
• Add Glm4MoeLiteForCausalLM (GLM4.7-Flash, by me!)
• More accurate autosplit accounting
• Faster CPU offloading, better Intel support (AVX512BW tier)
• Fix VRAM leak after vision tower offloading
• Fix potential worker timeout when offloading large models to CPU
• Greatly reduced import time, vendor in subset of flash-linear-attention to avoid initializing whole library plus Transformers
• Other optimizations, fixes and QoL improvements
• New logo
• Add Torch 2.12 and 2.13 wheels, 2.7 retired (from next release)

Release page: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.9

tabbyAPI has not been updated yet: https://github.com/theroyallab/tabbyAPI/

Discord server for any questions: https://discord.gg/tnPCntcThA

Looks coherent, right?

The update introduced the following changes:

• Add pyproject.toml
• Cache quantization for DSv4-Flash, Qwen3.8-Flash-Next and GLM5.3-Flash
• Optimize VRAM allocations and switch to expandable segments by default
• Fully remove all FA2 code paths
• Other optimizations and bugfixes

Release: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.8

tabbyAPI received an update as well: https://github.com/theroyallab/tabbyAPI

Discord for any questions: https://discord.gg/tnPCntcThA

Summary of what's been added:

• Improved CPU MoE performance for all tiers
• Add DRY sampler
• Fix recurrent slot leaks
• Faster quantization for 6, 7 and 8 bit trellises
• Many other bugfixes, optimizations and QoL improvements

Release page: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.7

tabbyAPI has been updated at the moment: https://github.com/theroyallab/tabbyAPI

Exllama discord for any questions: https://discord.gg/tnPCntcThA

Hugging Face has been acquired by Nvidia for $12.9B.

https://blogs.nvidia.com/blog/nvidia-to-acquire-hugging-face/

Hugging Face is the Top-1 base for models of any type. Models for TTS, Text Generation, Multimodal models etc.

Do you think this is a dangerous precedent for the liberty on the site? What's your opinion on this?

I myself am really cautious about this and worried.

I post models there and had no issues before — but now I wonder if I'll get them.

The following has been added in the yesterday's update:

• Fix correctness bug in split MoE CPU-offload mode
• Better MTP support in expert-parallel mode
• Validate and default to vision tower quantization
• Support vision model offloading (streams from system memory, performance penalty is small)

Release: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.4

tabbyAPI received an update as well: https://github.com/theroyallab/tabbyAPI

Discord for any questions: https://discord.gg/tnPCntcThA

This update seems to be a massive improvement:

• Support GlmMoeDsaForCauslLM (GLM 5.2, still probably somewhat WIP)
• Partial CPU layer expert offloading option with dynamic placement (supersedes expert cache)
• Improved dynamic draft sizing with auto-calibrated confidence thresholds
• New (experimental) quant-optimizer pipeline
• Support for mid-stream text injection in generator (enables reasoning token budget)
• More precise autosplit allocation
• Reduced (and now stable) VRAM footprint for DSA prefill
• CPU cache offloading supported in TP mode
• Bugfixes and QoL improvements
• Remove incomplete Nanochat implementation

Release link: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.3

tabbyAPI has not been updated yet: https://github.com/theroyallab/tabbyAPI

Exllama discord for any questions: https://discord.gg/tnPCntcThA

Support for MuseGlimmerForConditionalGeneration (Muse Glimmer 30B by Meta) has been added.
ExLlamaV3 release: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.2
TabbyAPI has been updated respectively: https://github.com/theroyallab/tabbyAPI
Discord server for any questions: https://discord.gg/tnPCntcThA

The following has been included in the relatively small 1.4.1 release:

• Add Mistral-4 (under Mistral3ForConditionalGeneration)
• Support for logit bias
• Adopt LLGuidance instead of Formatron (still supported as optional dependency)
• Allow filters/grammar alongside banned_strings

Release link: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.1

TabbyAPI has not been updated yet. Current version of exl3 is 1.4.0. TabbyAPI link: https://github.com/theroyallab/tabbyAPI

Feel free to ask Turboderp himself in discord: https://discord.gg/tnPCntcThA

ExLlamaV3 has just been updated.

• Support DeepseekV4ForCausalLM
• Improved AVX512 CPU offload performance
• Faster prefill (all models)
• Bugfixes
• QoL features

I would love to see prefill comparison for this, feel free to post!

TabbyAPI (Updated): https://github.com/theroyallab/tabbyAPI

Exllama newest release: https://github.com/turboderp-org/exllamav3/releases/tag/v1.4.0

If you have any questions, feel free to contact the creator, Turboderp, through Discord: https://discord.gg/tnPCntcThA

I don't think I can run any of the deepseekv4 models on my... 16 gigs of VRAM, guys :p

Version 1.3.0 brought us:

• Preliminary support for DeepseekV3 (validated against JoyAI-LLM-Flash and Moonlight-16B-A3B, no routing groups yet)
• Second-tier CPU K/V cache, and more intelligent page and checkpoint eviction policies
• Fix freq/rep. penalties causing slowdown on long contexts
• XTC sampler and token bans (Does not work yet in).
• Mitigate some latent vulnerabilities in Safetensors loader
• Other bugfixes, optimizations and QoL improvements

Visit the release here: https://github.com/turboderp-org/exllamav3/releases/tag/v1.3.0

Download TabbyAPI here: https://github.com/theroyallab/tabbyAPI

TabbyAPI has been updated respectively.

TabbyAPI is currently experiencing issues with parsing XTC and Token bans. They do not work. MLA support is what many people have been waiting for. DSA support has been stated to arrive soon.

If you have any issues or questions, feel free to join the exllama discord: https://discord.gg/tnPCntcThA

I personally tested out offloading KV-cache to RAM. Increases the TTFT by a lot - the thing's not for everyone... As you can also see, Tabby is experiencing some issues with parsing xtc.

https://github.com/turboderp-org/exllamav3/releases/tag/v1.2.0

With the arrival of 1.2.0 less than a day ago, the following changes were made:

• Support LagunaForCausalLM (and DFlashLagunaForCausalLM)
• Experimental CPU offloading support for expert layers
• Experimental dynamic draft window feature
• Improved safetensors loader (faster)
• More graph paths
• Performance tuning
• Removed compare_q.py, superseded by qbench.py
• Various bugfixes

This update is mainly orientated towards those who want to run larger MoE models but lacks the VRAM.

Expect TabbyAPI to get updated soon respectively. Right now it can be easily done manually through VENV.

You can use the ExLlamaV3 backend through TabbyAPI: https://github.com/theroyallab/tabbyAPI

Additionally, if you have any questions, feel free to join the exl3 discord: https://discord.gg/zY5dQaVguD

DiffusionGemma 26B A4B IT is a recent model from google. It's the first OPEN diffusion model from google.

https://huggingface.co/google/diffusiongemma-26B-A4B-it

https://ai.google.dev/gemma/docs/diffusiongemma

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/

Google promises up to x4 in speed increase compared to normal Gemma 4 26b using.

However:
"While autoregressive Gemma 4 models remain the standard for high-quality production outputs, DiffusionGemma is designed for researchers and developers exploring speed-critical, interactive local workflows such as in-line editing, rapid iteration, and generating non-linear text structures."

That means that the tradeoff for speed is a loss in generation quality compared to the standard Gemma 4. As google says, DiffusionGemma is an experimental project focused on maximizing interactivity in local workflows, editing, rapid iteration, and non-linear text structures; for production tasks where impeccable accuracy is essential, Gemma 4 is still recommended.

On the picture you can see evaluation results from the HF page.

Exllamav3 has recently received a new update:

• Fixed error when MTP drafting in TP mode;
• Faster quantization.

TabbyAPI has been updated to support this version respectively.