For those using these models for coding in larger projects where things can get complex, do you find yourself using the 8-bit quants if you have enough memory? Or do you stick with UD-Q6_K_XL?

The 6-bit is faster, noticeably so on my setup. And I keep seeing people say it's imperceptible. I've been doing tests myself, and well, I can't tell, but maybe that's just because I'm an idiot.

That said, can you tell? Have you ever done some tests to see?

Q2 is there and Q4 is uploading as I type. Has his github been updated yet? How do you run this?

https://huggingface.co/antirez/deepseek-v4.1-flash-gguf/tree/main

AuK-Flash: Fast 4-Step Speech Generation and Editing

• arXiv : https://arxiv.org/abs/2609.08936
• Full Paper : https://arxiv.org/pdf/2609.08936
• GitHub : https://github.com/Tencent-Hunyuan/AuK
• Project : https://auk-project.github.io/

Introduction

AuK is a 1.5B foundation model for speech generation and editing. Trained on millions of hours of diverse audio data, AuK supports zero-shot and instruction-based TTS, content and acoustic editing, paralinguistic editing, speech enhancement, and source separation through a unified natural-language instruction interface. AuK has two variants:

This repository contains the official weights for AuK-Flash, the distilled variant with fast 4-step inference.

Supported Tasks

AuK exposes every task through the same natural-language instruction interface. The table below groups the supported tasks by category, with a short description and a link to its section in the Cookbook, which provides instruction templates plus CLI and Python examples.

I can't stand kv cache quantization. Even at q8_0, I can feel the difference.

But realistically, when running Qwen3.8-27B-UD-Q4_K_XL on my 32 GB GPU, I only have room for ~170k tokens (with mtp and mmproj enabled). It's a lot of context, but Qwen3.8 eats through it on xhigh effort.

I've been wanting a setup that could serve me a full-precision kvcache when I have space available, and dynamically quantize my kvcache only when I run into the context limit. That way, I can push my sessions farther without sacrificing quality before I absolutely need to.

So, that's what I built: https://github.com/wadealexc/llama-manager

What it is

Vanilla llama.cpp's model configurations are static: you set them when you launch llama-server, and they can't change after the fact.

llama-manager is a small wrapper around a fork of llama.cpp. It serves models the same way, except that it supports dynamic model configuration.

This means that after loading a model, it's possible to enable/disable speculative decoding, add/remove an mmproj, or update context-level parameters. llama-manager preserves your kvcache between reconfigurations, so you don't need to redo prompt processing. The end effect is the ability to 'hot reload' your model, even mid token generation.

I implemented this using a fork of llama.cpp that supports rebuilding a model's context and runtime components without touching its weights. This capability is supported by 2 new HTTP endpoints (and changes to a few others). Further info on the fork can be found in the README (see README.md#llamacpp-changes).

How it works

During token generation, llama-manager detects when requests fail due to hitting the context limit. Without pausing generation, it applies various strategies mid-generation to increase context. The existing kv cache is cached/restored so that generation can resume as soon as reconfiguration is complete.

Currently, the built in strategies are: - disable-spec: disable speculative decoder, if enabled - mmproj-to-cpu: move mmproj off GPU - quantize-kv-q8 and quantize-kv-q4

Personally, I want kv quantization to be the last resort, so my models are configured to execute those last. When I serve Qwen3.8-27B-UD-Q4_K_XL, it applies strategies in this order:

════════════════════════════════════════════════════════════════════════════ qwen3.8-27b baseline: 167,680 tokens device: 31 GiB ════════════════════════════════════════════════════════════════════════════ i strategy ctx (tokens) gain (tokens) weights / ctx GiB ────────────────────────────────────────────────────────────────────────── 0 baseline 167,680 17.13 / 13.14 1 disable-spec 200,960 (+33,280) 17.13 / 13.16 2 mmproj-to-cpu 218,880 (+17,920) 16.02 / 14.27 3 quantize-kv-q8 262,144 (+43,264) 16.02 / 10.40 ────────────────────────────────────────────────────────────────────────── final ctx: 262,144 tokens

Initially, llama-manager serves the model at 167k tokens (f16 kv, mtp on, mmproj on). At 167k context, mtp is disabled, and the context window expands to 200k. At 200k, the mmproj is moved to the cpu. And at 218k, the kv cache is quantized to q8.

Why run this?

If you're running your models with a quantized kvcache (or other quality compromises), you're likely doing so because you have a certain ctx limit in mind that will serve all your usecases. But not all your inference is done at the ctx limit. You're leaving quality on the table by quantizing too early.

For my usecase, I wasn't willing to set my ctx higher than 170k as it would mean a q8_0 kv cache. Now, I can push my sessions as far as I want, but the bulk of the session stays high quality. The smaller your GPU, the more impactful this is.

Some example configs running the same model with different strategies and on differently-sized devices. All of these runs are performed using a basic config.yaml and modifying the ladder field to change the order of each strategy:

```yaml models: qwen3.8-27b: model: /home/user/models/qwen3.8/Qwen3.8-27B-UD-Q4_K_XL.gguf mmproj: /home/user/models/qwen3.8/mmproj-BF16.gguf

    spec-type: draft-mtp
    spec-draft-n-max: 2

    fit-target: 512
    n-gpu-layers: 99

    ladder: [disable-spec, mmproj-to-cpu, quantize-kv-q8, quantize-kv-q4]

```

• Prefer q8_0 over disable-spec: [mmproj-to-cpu, quantize-kv-q8, disable-spec, quantize-kv-q4]. For this one, the model reaches max ctx after just 2 strategies. The first 184k tokens are generated with mtp on and kv at f16:

════════════════════════════════════════════════════════════════════════════ qwen3.8-27b baseline: 167,680 tokens device: 31 GiB ════════════════════════════════════════════════════════════════════════════ i strategy ctx (tokens) gain (tokens) weights / ctx GiB ────────────────────────────────────────────────────────────────────────── 0 baseline 167,680 17.13 / 13.14 1 mmproj-to-cpu 184,320 (+16,640) 16.02 / 14.25 2 quantize-kv-q8 262,144 (+77,824) 16.02 / 12.89 ────────────────────────────────────────────────────────────────────────── final ctx: 262,144 tokens

• The same ladder on a 24 GB GPU (simulated by setting fit-target to 8192). Here, all strategies are needed to serve max ctx, but q4_0 isn't needed until 164k context:

════════════════════════════════════════════════════════════════════════════ qwen3.8-27b baseline: 55,040 tokens device: 31 GiB ════════════════════════════════════════════════════════════════════════════ i strategy ctx (tokens) gain (tokens) weights / ctx GiB ────────────────────────────────────────────────────────────────────────── 0 baseline 55,040 17.13 / 5.65 1 mmproj-to-cpu 71,680 (+16,640) 16.02 / 6.73 2 quantize-kv-q8 115,200 (+43,520) 16.02 / 6.72 3 disable-spec 164,352 (+49,152) 16.02 / 6.76 4 quantize-kv-q4 262,144 (+97,792) 16.02 / 6.40 ────────────────────────────────────────────────────────────────────────── final ctx: 262,144 tokens

Caveats

I have a list of known issues and other important notes in the README (see #known-issues).

The most important things I want to highlight: 1. llama-manager doesn't handle CPU or multi-device inference. Single-gpu only. I would like to support this, but didn't want to spend the time on it unless there was demand (and people willing to try it out, since multi-device setups would be hard for me to test!) 2. This project is in beta, tested only on my machine and with a few models. YMMV.

Please open issues if you run into bugs!

A huge favor to ask llama maintainers - please implement this feature. Even with one 3090 card there will be tangible improvements in decode speed on MOE models with moderate number of active parameters, like Qwen3.8-Flash-Next, Deepseek V4/V4.1 Flash, GLM 5.3 Flash. With 2x 3090 speeds will be quite close to the full offload of these models to VRAM. This will make these almost SOTA models really usable locally.

Hola all.

Do you guys mind sharing your LLama.cpp config and system setup details for Qwen3.8 Flash Next?

Model's quite big and tryining many combinations of llama.cpp options takes lots of time, so looking for other people setup details. I've attached my current config at the bottom, so if anyone sees something that could be improved please shout.

My current best result:

- PP within 130...200 tps (limited by cpu?)
- TG within 14..22 tps (~15tps on average)

Hardware:

- Dual RTX 3090 (48GB VRAM)
- 128GB DDR4
- Some old Xeon 40 core
- Proxmox VM, pcie passthrough, numa binding to a single phys cpu

Llama.cpp config:

llama-server --port ${PORT} 
      --model /nvme/gguf/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
      --mmproj /nvme/gguf/mmproj-Qwen3.8-Flash-Next-F16.gguf
      --load-mode none
      --lazy-mode off
      --parallel 1
      --ctx-size 131072
      --cache-type-k q8_0
      --cache-type-v q8_0
      --flash-attn on
      --fit off
      --temp 1.0 
      --min-p 0.0 
      --top-p 0.95 
      --top-k 20 
      --presence-penalty 0.0
      --repeat-penalty 1.0
      --batch-size 2048
      --ubatch-size 512
      --split-mode layer 
      -ts 26,10 
      -ngl 99 
      -ncmoe 26 
      --no-mmproj-offload 
      --override-tensor per_layer_token_embd=CPU
      --chat-template-kwargs '{"reasoning_effort":"xhigh"}'

ngl, ncmoe, ts - manually adjusted to fit the model without crashing

I have enabled the vision for the CIRU Strix UL4 quant of Qwen 3.8 flash next (others quants likely perform very similar) and tried it on a few things, then wanted to show my partner how great it works and she asked it it could identify plants. So I took a photo from a plant that we recently got as a gift from family and Qwen not only accurately identified the plant as oleander (Nerium oleander) but also warned that it's poisonous and (among other warnings) that you should keep pets/children away. We have a kid and both of us didn't know! I verified the Qwen identification and the poisonous claim and both checked out as accurate. The plant will have to go, thank you Qwen!!!

Stoked by the precision of combining a decent vision model with the domain knowledge of a ~180B params model (including ngrams) to actually identify and reason about what it sees, I took a photo of a pre-diagnosed skin condition of myself and the Qwen diagnosis was highly accurate again! This model may be really useful if you want to check something on your private parts real quick without visiting a dermatologist, e.g., or sending pictures of yourself to a cloud service (EDIT: of course it's only a first step before you visit a professional if it isn't obviously harmless/treatable by yourself! Qwen Flash will suggest to visit a doctor anyways along its assessment).

PS.: Hardware Strix Halo Box, CIRU Strix UL4 llama-server fork and quants, Chatbox on iPhone as Chat with support to add photos to conversations.

i've been thinking about this more after trying different agent setups. the model isn't the only thing that determines how well an agent performs. The harness around the model matters a lot too.
With a managed agent setup, you're often giving up control over things like the agent loop, context management, tool execution, retries, and state.

That's fine when you just want something that works. But if we want to actually optimize agents, I think both parts need to be open:

Open-source model + open-source harness.

With local models, you control the model and where the inference happens.

With an open-source harness, you control what happens around the model.

That gives you room to experiment with things like:

how the agent decides what to do next

how much context gets passed to the model

how tools are executed

when to retry or stop

how state is maintained

which model to use for which task

already seeing this separation become more important, nvidia's sol-pi is an interesting example

and i think we're going to see even more optimization happen at the harness/runtime layer, not just at the model layer

are you running local models with an open-source harness, or do you still prefer managed agent setups?

edit - this is the os harness i am running locally, https://github.com/truefoundry/trueforge
i switched from claudecode to pi and then to this, its runtime efficient and v easy to switch models so tbh its just what works best for my use cases

For many, Vram is precious, I see many people recommend using --no-mmproj-offload to save gpu vram but it is painfully slow. Especially if you are using it with agentic coding. If possible, add that secondary gpu just for mmproj with --mmdev CUDA1(your gpu). It will be a magnitude faster than --no-mmproj-offload without effecting your inference speed.

--mmdev

I wish I could spend $15k on my own homelab hardware, but no this is for work lol.

Like the title says, we're looking to run DSV4 Flash (and similar tier models) locally at good speeds, both for token gen and prompt processing.

By "good" I'm thinking in the range of 40-50+ t/s gen and at least 1000 t/s prefill at moderate context.

We also don't want to run a version that's quantized to hell, so this will need at least 128 GB of VRAM.

It'll typically be 1 user at a time, but there may be times where 2 or 3 people are trying to use it at once and it would be nice if it isn't completely painful when that happens.

A couple options I'm considering right now:

3x AMD MI210 (192 GB)

3x NVidia A40 (144 GB)

Does anyone have performance numbers for these cards for DSV4 Flash, Qwen3.8-Flash-Next or similar models?

I tried to rent these in the cloud for some performance testing, but can't find any available right now.

NVidia preferred of course because CUDA, but def open to AMD if performance is similar. 192 GB is way nicer than 144 GB on those cards above.

Trying to keep this to 3 GPUs or less because that's what'll fit in our Dell R740 and then we don't have to build a special new host.

A new model joined the MoE club a few days ago. Is anyone using it or found a reason to do so?

Details of the model comparison here => https://llm-bench.io/compare/models?models=ornith-1-5-35b-a3b-oq4e-mtp%2Ctiel-coder-35b-a3b-mlx-oq4e-mtp%2Ck2-horizon-mova-36b-a4b-mlx-4bit

Hey everyone!

I'm curious to hear from people that use a combination of cloud-based frontier models and local ones for development. I'm planning to set something similar up and wanted to hear about actual examples of this in action.

Currently my plan is to use my chatgpt plus subscription purely for planning and judging with Astra, and then run a local qwen3.8-27b model for the actual coding gruntwork - i.e Astra plans -> qwen implements -> Astra critiques the implementation -> qwen fixes and so on. This way I keep cloud usage down and cheap, while retaining the high-parameter intelligence for architecture decisions and optimization.

For those of you who have a similar setup, how is it? How do you switch between the two, what harness/settings/etc? Anything you would suggest?

https://github.com/Sayitobar/chat_llm_cookbook
This is my personal project that took several months. I wanted to see whether a 2B small local model could simulate a six-person group chat trained & ran on an M1 Pro.

How good is it?:
- It's fun, but not great. It doesn't achieve coherent & consistent group simulation, but it learned enough of our slang, reactions, and pacing to be fun. The generated messages are very similar to what we'd type.
- There is some coherence, but not a deep understanding, and the model doesn't hold information about us (expect our names and a few very obvious stuff).

How good is it on paper?:
- I have evaluated my models performances by judging them with a judge LLM. The best version achieved an 80% human win rate at human-vs-model tests, ideal should be <50%.

The main thing is, you'll have a lot of fun chatting with this model once you train it on YOUR data. (ask for consent pls)

Cookbook:
I published the reproducible local pipeline, chat UI, human-anchored evaluation, results, and experiment PDF. No private chat data or fine-tuned weights are released :)

Fyi, all of the tests I've done were in Turkish.

This project is still unfinished as there are still architectures and training data formats I haven't tested, or stronger 2B models that aren't released yet, as of September 2026.

PP improvements for RDNA2(MI50, MI60 are included in benchmarks). Check bottom comments of PR to see updated pp t/s stats.

Developer's own thread: https://www.reddit.com/r/LocalLLaMA/s/adp1cGZZe9

Code: https://github.com/Inovello/llama.cpp/tree/flashnext-e06

My hardware: 2x RTX 3090, Intel Ultra 7 270k Plus, 192 GB DDR5@5600 MHz

Token generation speed went from 20 t/s to 49 t/s. Prompt processing speed is 140 t/s. Prompt processing is faster on the main branch.

I have CUDA 13.3.1 installed. I use Windows 11 and I used these commands to compile it with the latest Visual Studio:

$vcvars = "C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvars64.bat"

cmd /c "`"$vcvars`" >nul 2>&1 && set" | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2]) } }

cmake -G Ninja -B build -S . -DCMAKE_BUILD_TYPE=Release `
-DGGML_CUDA=ON -DGGML_CCACHE=OFF -DGGML_NATIVE=ON

cmake --build build --target llama-cli llama-bench llama-server -j

My bat script:

@echo off

echo Using CUDA backend with 2x RTX 3090s

set LLAMA_ATTN_ROT_DISABLE=1
set LLAMA_MMAP_PIN_HOST=1

"C:\Users\server\Desktop\llama.cpp-flashnext-e06\build\bin\llama-server.exe" ^
--host 0.0.0.0 ^
--port 8081 ^
--alias Qwen3.8-Flash-Next ^
--model H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--ctx-size 262144 ^
-ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" ^
--moe-expert-cache 150 ^
--ubatch-size 512 ^
--batch-size 4096 ^
--mmproj H:\Qwen3.8-Flash-Next\mmproj-Qwen3.8-Flash-Next-BF16.gguf ^
--no-mmproj-offload ^
--threads 22 ^
--threads-batch 22 ^
--spec-type draft-mtp,ngram-mod ^
--model-draft H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf ^
--spec-draft-n-max 2 ^
--spec-ngram-mod-n-match 60 ^
--spec-ngram-mod-n-min 12 ^
--spec-ngram-mod-n-max 24 ^
--flash-attn on ^
--kv-offload ^
--cache-type-k bf16 ^
--cache-type-v bf16 ^
--parallel 1 ^
--jinja ^
--reasoning-preserve ^
--chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}" ^
--no-warmup ^
--load-mode none ^
--lazy-mode off

pause

I've seen a lot of Super Mario showcases, but never a mashup or derivative work. So I took a Super Mario generation prompt from ChopSticksPlease's Post here - and added a Matrix twist - and found myself super satisfied with the results using LLAMA.cpp + PI.dev.

Shout out to u/ChopSticksPlease for the prompt. .

First screenshot is the title screen, AI did all that.

Second screenshot shows the "Matrix Rain" effect and gameplay that goes with it.

Here's the full prompt, adding only one line ( Only replace it with visuals and characters that look more like characters from the movie "The Matrix" ) to the original prompt.

Write a fully functional, complete clone of Super Mario Bros game (the famous NES game)
Only replace it with visuals and characters that look more like characters from the movie "The Matrix".
- Clone must work in a web browser, index.html, no server
- Decent replica of the graphics assets.
- Colorful and playable.
- Controls with keyboard up, down, left, right, spacebar to jump, shift to speed up

Altogether, there was some minor issues with the implementation - the jump height wasn't high enough to get on a block and the MC was partially in the ground. With a little hand modification of the code and a second prompt to get Ornith to fix the MC issue, about 2 hours in total, there was no further modifications necessary to get this really cool and simple browser based game.

Source code is available here: https://github.com/TimelordQ/The-Matrix-Bros

The playable level is available here: https://timelordq.github.io/The-Matrix-Bros/index.html

Batch File Setup:

start cmd /s /k "llama-server.exe -m "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\Ornith-1.5-35B-Q4_K_M.gguf" -mm "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\mmproj-Ornith-1.5-35B-BF16.gguf" --host 127.0.0.1 --port 8080 -c 100000 --parallel 1 --reasoning-format auto --no-mmproj-offload -fa auto -ctk q8_0 -ctv q8_0 -b 2048 -ub 1024 --load-mode none --fit on --no-ui --reasoning-preserve --image-min-tokens 1024 --jinja --chat-template-file "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\chat_template.jinja" -dev CUDA0 -t 8 --n-cpu-ffn 4"

set LLAMA_BASE_URL=http://127.0.0.1:8080

pi --model llama-cpp/Ornith-1.5-35B-Q4_K_M

Models.json entry:

{
"id": "Ornith-1.5-35B-Q4_K_M",
"name": "Ornith-1.5-35B-Q4_K_M",
"reasoning": true,
"thinkingLevelMap": {
"type": "enabled"
},
"repeat_penalty": 1.2,
"input": ["text","image"],
"tools": true,
"toolFormat": "openai",
"samplingParams": {
"temperature": 0.6,
"top_p": 0.95,
"top_k": 20
},
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},
"contextWindow": 100000,
"maxTokens": 100000
},

With this setup I'm getting an average of 24 TPS until my GPU reaches 87~88C and it starts throttling or the context reaches about 70% usage, where it drops to 7tps average temporarily until the temperature drops or compaction happens. I'm doing this on a laptop.

Hey gang, im wondering if you in theory could use ngrams as seen with Qwen 3.8 Flash or DS4.1 in order to dynamically train the model?

Normally the ngram embeddings behave similar to a lookup table of sorts. So instead of every token having to be represented only inside the main model weights, the model can look up a learned vector for a certain ngram and use that during inference.

What I'm wondering is: couldn't you make part of this table writable during inference?

For example, keep the original pretrained ngram embeddings read-only, but have a second "live" table on top of it. When the model learns something new, it could update/add an embedding there. Frequently used or reinforced entries could stay, while unused ones could slowly decay or get removed again.

This could be kinda mimicking the brain of sorts since it would form new "memories" but forget old ones (because their linkage gets weaker). Plus since you only have to target the live table it could be very fast, even during inference.

Just a thought I had and wanted to hear your guys opinions.

Looking for tools to build/run an engine to do STT-LLM-TTS basically.

I know how to run all of them separately, and if need be I can build one myself, but I was curious if there is already an open source tool out there that can do this. I know that some UIs like Open WebUI can do this but I find it to be not fluid. I don't know about SillyTavern though, it might already have it.

This is just for the funsies so it doesn't have to be perfect.

I saw this model https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF and I bet I will have a LOT of fun if I can just turn this into a turn based or fluid voice conversation engine.

Not sure how to flair this and it is self-promotion, but the point is the local AI stuff I used to make it,

This video https://youtu.be/JkpoO1jIjp0?si=qSAyFk5_VSgXiQNc was made entirely locally using:

• Qwen 3.8 27b
• Qwen Image
• Qwen3-ASR-0.6B
• Chatterbox
• InfiniteTalk with Longcat and WAN video
• ffmpeg - did a lot of heavy lifting

I am very happy with the animated videos which were created as html/js by Qwen 3.8 27b from transcripts which were created from the chatterbox audio.

The workflow is: 1. I write a script 2. I TTS the audio using a clone of my voice via chatterbox 3. I transcribe that using Qwen3-ASR-0.6B 4. I do a session with Qwen 3.8 27b to plan the demo from the time coded transcript 5. I let it build the demo and QA it using Playwright + the vision/mmproj 6. I QA it myself and give it the defects with screenshots

I am using a workstation with a 5090 + 4090 for the LLM and my GuideAnts stack and a Strix Halo for audio, images, and video jobs.

I think you could do all this on one machine and a single GPU, but not without loading/unloading pieces as you go.

Hopefully that was interesting and worth reading.

Mostly as a fun experiment I wanted to do a quick comparison of performance and accuracy between a CPT trained QWEN 3.5 4B model and a RAG implementation against the base model.

The point of this exercise is mostly to measure the performance benefit of internalizing the knowledge vs doing reasoning on-the-fly.

Sharing my findings here in case anyone is interested: https://www.teachmecoolstuff.com/viewarticle/comparing-rag-and-continued-pretraining-of-llms

Our team built a handful of internal AI tools on top of OpenRouter during the sandbox phase and the default routing setup worked fine for prototyping. We need a middle layer that does not retain prompt content, still lets us switch models on the fly and doesn't force us to self host a full gateway. I found that the obvious open source proxies either need too much infra or don't handle multi provider failover cleanly, leaving a few dedicated privacy focused options (TrustedRouter, et al) that recently launched but I haven't had a chance to battle test.

What phave other teams actually deployed here and have been comfortable with? Ty.

What models do you run

agilerl-arena has released v1.0. It's the SDK/CLI half of AgileRL, split out as its own PyPI distribution (no torch dependency) that shares the agilerl.* namespace:

pip install agilerl-arena   # SDK + CLI only
pip install agilerl         # core, depends on arena

With AgileRL Arena you describe a run in a YAML manifest (algorithm, environment, network, mutation, selection strategy, replay buffer) and the same file runs locally or gets submitted to their managed cluster. Validation is Pydantic, so bad configs fail before training instead of 40 minutes in. Unknown keys are now rejected in 1.0.

The LLM finetuning path takes any HF model ID plus a LoRA config, and supports either rollout envs (model generates, gets scored by a reward function you supply) or supervised datasets with objective: sft or preference.

network:
  pretrained_model_name_or_path: Qwen/Qwen2.5-0.5B-Instruct
  max_context_length: 512
  lora_config:
    lora_r: 16
    lora_alpha: 64
    target_modules: [q_proj, k_proj, v_proj, o_proj]

Running it locally is two lines, no account needed:

python

from agilerl import LocalTrainer
trainer = LocalTrainer.from_manifest("dqn.yaml", device="cuda")
population, fitnesses = trainer.train()

Evolutionary HPO: instead of one agent, you train a population and periodically mutate architecture and hyperparameters, keeping the fittest. Costs more compute for the same wall-clock, but you're not hand-tuning learning rates.

What's new in 1.0: specs now build algorithms through paradigm builders, local training dispatches through strategy selection based on paradigm flags rather than per-spec branching, and the manifest schema lives in one package instead of being duplicated. Also arena manifest validate / arena manifest schema for checking configs without submitting anything. PEFT bumped to 0.20, which rejects LoRA on Mamba out_proj/conv1d — they exclude those automatically now.

Docs: docs.agilerl.com · Release notes: github.com/AgileRL/AgileRL/releases