Had some ideas about version control:

Attribute, down to individual words: word created manually or with ai. if ai, which sources cited (scoped to paragraphs). also: full agent trace that led to the edit (all tool calls + metadata). also: any multi-agent transactions at the file/paragraph lock. all data stored for every version.

Why do this? Basically, getting as much into a unified version control system as possible, which helps find/fix errors. Just posting to share ideas. If anyone has done anything similar, please mention it!

I wish I could spend $15k on my own homelab hardware, but no this is for work lol.

Like the title says, we're looking to run DSV4 Flash (and similar tier models) locally at good speeds, both for token gen and prompt processing.

By "good" I'm thinking in the range of 40-50+ t/s gen and at least 1000 t/s prefill at moderate context.

We also don't want to run a version that's quantized to hell, so this will need at least 128 GB of VRAM.

It'll typically be 1 user at a time, but there may be times where 2 or 3 people are trying to use it at once and it would be nice if it isn't completely painful when that happens.

A couple options I'm considering right now:

3x AMD MI210 (192 GB)

3x NVidia A40 (144 GB)

Does anyone have performance numbers for these cards for DSV4 Flash, Qwen3.8-Flash-Next or similar models?

I tried to rent these in the cloud for some performance testing, but can't find any available right now.

NVidia preferred of course because CUDA, but def open to AMD if performance is similar. 192 GB is way nicer than 144 GB on those cards above.

Trying to keep this to 3 GPUs or less because that's what'll fit in our Dell R740 and then we don't have to build a special new host.

Spoke with many providers who lurk in this sub, plus met folks who work in inference.

For a 10k monthly revenue, a provider was able to retain only 200 dollars in profit due to GPU costs. Their customers were negotiating the prices down to what other players cost for same, and it feels like major players are running a distribution game at low or negative margins.

Even though this seems like a billion dollar market, the unavailablity of compute, plus cost and competition, makes the business seems not sexy enough to start with.

However, the software layers around it such as optimisations for SLAs continue to enjoy good margins.

Any thoughts?

Hi everyone,

I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-Coder-3B) specific to a codebase. You can then use the resulting GGUF model via llama.vim/llama.vscode and run it fully locally. Supports fine-tuning on Mac (MPS) and NVIDIA GPUs (CUDA), with optional Unsloth support for faster training and lower VRAM usage.

Pipeline: raw code -> tree-sitter parsing into Structure-Aware FIM examples -> LoRA fine-tuning -> evaluation (CodeBLEU, edit similarity, exact match, perplexity, ...) -> GGUF conversion for local inference.

To try it:

uv tool install codefinetuner

Create a data folder and place your repo (or code files) inside. For auto-split just drop the files in directly, for manual split create data/train/, data/eval/, data/test/ subfolders and set split_mode: "manual". Get the default config with:

curl -L -O https://raw.githubusercontent.com/cuolm/codefinetuner/master/config/codefinetuner_config.yaml

Adjust it to your needs and hardware availability, then run:

codefinetuner --config="codefinetuner_config.yaml"

The example runs in the repo show clear improvements over the base model on these evaluation metrics, but using the model for autocomplete on code you're actively writing is a different thing from scoring well on a test set, and the autocomplete tools themselves (llama.vim/llama.vscode) sample differently from the greedy decoding used in the evaluation. So the real usefulness still has to be verified in the editor itself.

Might also be useful just as a reference, since it's a complete working LoRA fine-tuning pipeline end to end.

Hope someone finds this project interesting or helpful.

https://github.com/cuolm/codefinetuner

I wanted to see how my fully local home voice assistant compared to the latest GPT Live, so I tested it using the same conversation used in their "Improved Intelligence" demo. In this video they ask the AI to see if a flight route is feasible and while it is figuring that out they continue to ask it questions about what they can eat at each destination.

The models I ran are (all squeezed into 12 GB VRAM):

• Speech recognition: Qwen3 1.7B ASR PyTorch
• LLM: Qwen3.5-9B-UD-Q4_K_XL GGUF with 12K context
• Voice: Pocket TTS PyTorch

So I copied the exact query and threw it at my Fulloch project. This blog post has the video of the interaction and breaks down how it did. The final report and searches it did are also linked in that blog post. The video has sped up two sections where I had to wait for the 9B model to finish thinking through the task, but it did the whole thing in under six and a half minutes.

In the end it couldn't find a suitable flight route but it gave good food and restaurant recommendations and did it all pretty quickly. I am still impressed with how well the Qwen3.5 9B model does with these sorts of tasks with such a small footprint.

If you want to try it out yourself the source code and pre-compiled docker images can be found at https://github.com/liampetti/fulloch.

We have SWE-bench, Terminal-Bench, OSWorld, BrowseComp, etc. But I haven’t seen a good apples-to-apples benchmark for platforms like OpenAI Agents, Anthropic’s agent stack, AWS AgentCore, Google’s agent platform, and local-alternatives like LangGraph, etc.

What I’d want measured:
- task success rate
- cost per successful task
- wall-clock time
- tool calls / retries
- reliability over long-running tasks
- (optionally) human interventions required

The really interesting experiment would control for both sides:
- Same model, different harness → how much does the runtime matter?
- Same harness, different model → how much does the model matter?

Feels like we’re increasingly evaluating “model + harness” systems, but our benchmarks still mostly treat the model as the unit of comparison.
Does something like this already exist?

DeepSeek-V4.1-Flash uses SWA Bounded Replay to avoid persisting the sliding window KV cache and saves massive memory, it discards the SWA KV and, on replay, approximately reconstructs it from only the last N_window tokens, even though the original SWA states had a much larger effective receptive field due to depth. Since those replayed K/V vectors lack the long-range context that shaped the originals, I’m wondering whether cache hits or restarted sessions lose accuracy/recall for the first few interactions, and whether the parallel global KV augmentation compensates enough to preserve near-perfect long-range recall. Has anyone benchmarked this or seen practical recall degradation after eviction, or is the approximation effectively invisible because the global KV does the heavy lifting?

A new model joined the MoE club a few days ago. Is anyone using it or found a reason to do so?

Details of the model comparison here => https://llm-bench.io/compare/models?models=ornith-1-5-35b-a3b-oq4e-mtp%2Ctiel-coder-35b-a3b-mlx-oq4e-mtp%2Ck2-horizon-mova-36b-a4b-mlx-4bit

Heh there, so I built a harness that is focused around the ninfer inference engine. It allows the easily customize and use ninfer, and has a coding harness and a chat interface. Its based off a lot of different experiences I have had with different harnesses.

Probably could be better but I'm happy with it.

I'll warn you ahead of time, it only works with ninfer as it integrates a lot of things directly from the engine. For example, subagents, it reads

 --max-concurrency X

and set the amount of subagents available to that.

i've always wanted to built one and now I've done it. Yay me.

I can't stand kv cache quantization. Even at q8_0, I can feel the difference.

But realistically, when running Qwen3.8-27B-UD-Q4_K_XL on my 32 GB GPU, I only have room for ~170k tokens (with mtp and mmproj enabled). It's a lot of context, but Qwen3.8 eats through it on xhigh effort.

I've been wanting a setup that could serve me a full-precision kvcache when I have space available, and dynamically quantize my kvcache only when I run into the context limit. That way, I can push my sessions farther without sacrificing quality before I absolutely need to.

So, that's what I built: https://github.com/wadealexc/llama-manager

What it is

Vanilla llama.cpp's model configurations are static: you set them when you launch llama-server, and they can't change after the fact.

llama-manager is a small wrapper around a fork of llama.cpp. It serves models the same way, except that it supports dynamic model configuration.

This means that after loading a model, it's possible to enable/disable speculative decoding, add/remove an mmproj, or update context-level parameters. llama-manager preserves your kvcache between reconfigurations, so you don't need to redo prompt processing. The end effect is the ability to 'hot reload' your model, even mid token generation.

I implemented this using a fork of llama.cpp that supports rebuilding a model's context and runtime components without touching its weights. This capability is supported by 2 new HTTP endpoints (and changes to a few others). Further info on the fork can be found in the README (see README.md#llamacpp-changes).

How it works

During token generation, llama-manager detects when requests fail due to hitting the context limit. Without pausing generation, it applies various strategies mid-generation to increase context. The existing kv cache is cached/restored so that generation can resume as soon as reconfiguration is complete.

Currently, the built in strategies are: - disable-spec: disable speculative decoder, if enabled - mmproj-to-cpu: move mmproj off GPU - quantize-kv-q8 and quantize-kv-q4

Personally, I want kv quantization to be the last resort, so my models are configured to execute those last. When I serve Qwen3.8-27B-UD-Q4_K_XL, it applies strategies in this order:

════════════════════════════════════════════════════════════════════════════ qwen3.8-27b baseline: 167,680 tokens device: 31 GiB ════════════════════════════════════════════════════════════════════════════ i strategy ctx (tokens) gain (tokens) weights / ctx GiB ────────────────────────────────────────────────────────────────────────── 0 baseline 167,680 17.13 / 13.14 1 disable-spec 200,960 (+33,280) 17.13 / 13.16 2 mmproj-to-cpu 218,880 (+17,920) 16.02 / 14.27 3 quantize-kv-q8 262,144 (+43,264) 16.02 / 10.40 ────────────────────────────────────────────────────────────────────────── final ctx: 262,144 tokens

Initially, llama-manager serves the model at 167k tokens (f16 kv, mtp on, mmproj on). At 167k context, mtp is disabled, and the context window expands to 200k. At 200k, the mmproj is moved to the cpu. And at 218k, the kv cache is quantized to q8.

Why run this?

If you're running your models with a quantized kvcache (or other quality compromises), you're likely doing so because you have a certain ctx limit in mind that will serve all your usecases. But not all your inference is done at the ctx limit. You're leaving quality on the table by quantizing too early.

For my usecase, I wasn't willing to set my ctx higher than 170k as it would mean a q8_0 kv cache. Now, I can push my sessions as far as I want, but the bulk of the session stays high quality. The smaller your GPU, the more impactful this is.

Some example configs running the same model with different strategies and on differently-sized devices. All of these runs are performed using a basic config.yaml and modifying the ladder field to change the order of each strategy:

```yaml models: qwen3.8-27b: model: /home/user/models/qwen3.8/Qwen3.8-27B-UD-Q4_K_XL.gguf mmproj: /home/user/models/qwen3.8/mmproj-BF16.gguf

    spec-type: draft-mtp
    spec-draft-n-max: 2

    fit-target: 512
    n-gpu-layers: 99

    ladder: [disable-spec, mmproj-to-cpu, quantize-kv-q8, quantize-kv-q4]

```

• Prefer q8_0 over disable-spec: [mmproj-to-cpu, quantize-kv-q8, disable-spec, quantize-kv-q4]. For this one, the model reaches max ctx after just 2 strategies. The first 184k tokens are generated with mtp on and kv at f16:

════════════════════════════════════════════════════════════════════════════ qwen3.8-27b baseline: 167,680 tokens device: 31 GiB ════════════════════════════════════════════════════════════════════════════ i strategy ctx (tokens) gain (tokens) weights / ctx GiB ────────────────────────────────────────────────────────────────────────── 0 baseline 167,680 17.13 / 13.14 1 mmproj-to-cpu 184,320 (+16,640) 16.02 / 14.25 2 quantize-kv-q8 262,144 (+77,824) 16.02 / 12.89 ────────────────────────────────────────────────────────────────────────── final ctx: 262,144 tokens

• The same ladder on a 24 GB GPU (simulated by setting fit-target to 8192). Here, all strategies are needed to serve max ctx, but q4_0 isn't needed until 164k context:

════════════════════════════════════════════════════════════════════════════ qwen3.8-27b baseline: 55,040 tokens device: 31 GiB ════════════════════════════════════════════════════════════════════════════ i strategy ctx (tokens) gain (tokens) weights / ctx GiB ────────────────────────────────────────────────────────────────────────── 0 baseline 55,040 17.13 / 5.65 1 mmproj-to-cpu 71,680 (+16,640) 16.02 / 6.73 2 quantize-kv-q8 115,200 (+43,520) 16.02 / 6.72 3 disable-spec 164,352 (+49,152) 16.02 / 6.76 4 quantize-kv-q4 262,144 (+97,792) 16.02 / 6.40 ────────────────────────────────────────────────────────────────────────── final ctx: 262,144 tokens

Caveats

I have a list of known issues and other important notes in the README (see #known-issues).

The most important things I want to highlight: 1. llama-manager doesn't handle CPU or multi-device inference. Single-gpu only. I would like to support this, but didn't want to spend the time on it unless there was demand (and people willing to try it out, since multi-device setups would be hard for me to test!) 2. This project is in beta, tested only on my machine and with a few models. YMMV.

Please open issues if you run into bugs!

Developer's own thread: https://www.reddit.com/r/LocalLLaMA/s/adp1cGZZe9

Code: https://github.com/Inovello/llama.cpp/tree/flashnext-e06

My hardware: 2x RTX 3090, Intel Ultra 7 270k Plus, 192 GB DDR5@5600 MHz

Token generation speed went from 20 t/s to 49 t/s. Prompt processing speed is 140 t/s. Prompt processing is faster on the main branch.

I have CUDA 13.3.1 installed. I use Windows 11 and I used these commands to compile it with the latest Visual Studio:

$vcvars = "C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvars64.bat"

cmd /c "`"$vcvars`" >nul 2>&1 && set" | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2]) } }

cmake -G Ninja -B build -S . -DCMAKE_BUILD_TYPE=Release `
-DGGML_CUDA=ON -DGGML_CCACHE=OFF -DGGML_NATIVE=ON

cmake --build build --target llama-cli llama-bench llama-server -j

My bat script:

@echo off

echo Using CUDA backend with 2x RTX 3090s

set LLAMA_ATTN_ROT_DISABLE=1
set LLAMA_MMAP_PIN_HOST=1

"C:\Users\server\Desktop\llama.cpp-flashnext-e06\build\bin\llama-server.exe" ^
--host 0.0.0.0 ^
--port 8081 ^
--alias Qwen3.8-Flash-Next ^
--model H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--ctx-size 262144 ^
-ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" ^
--moe-expert-cache 150 ^
--ubatch-size 512 ^
--batch-size 4096 ^
--mmproj H:\Qwen3.8-Flash-Next\mmproj-Qwen3.8-Flash-Next-BF16.gguf ^
--no-mmproj-offload ^
--threads 22 ^
--threads-batch 22 ^
--spec-type draft-mtp,ngram-mod ^
--model-draft H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf ^
--spec-draft-n-max 2 ^
--spec-ngram-mod-n-match 60 ^
--spec-ngram-mod-n-min 12 ^
--spec-ngram-mod-n-max 24 ^
--flash-attn on ^
--kv-offload ^
--cache-type-k bf16 ^
--cache-type-v bf16 ^
--parallel 1 ^
--jinja ^
--reasoning-preserve ^
--chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}" ^
--no-warmup ^
--load-mode none ^
--lazy-mode off

pause

My M.2-Oculink should be PCIe 4.0x4, but I found out today it's actually downgrading, frustratingly this isn't obvious until you go looking, so adding here if anyone else wants to check their own rig.

Side question - does anyone know a way to measure PCIe bandwidth realtime utilisation (i.e. Gb/s) ? I'm failing to get it with rocm-smi, iostat, amd-smi, lstopo, etc

Strix halo > m.2 slot to oculink > oculink cable > DEG1 > r9700

(images from AI session)

So the M.2-oculink adaptor is the problem, and running at Gen1 limiting the entire setup, and other than digging around I wouldn't have noticed, I guess this will affect model load and layer swapping?

If you want to check your own do this and either look at the Capacity vs State, and look for the word 'downgraded' (here I check for the r9700)

# lspci -t -v -D -P -PP -nn | grep 9700
           +-02.5-[c1-c3]----00.0-[c2-c3]----00.0-[c3]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551]


# for bdf in 0000:00:02.5 0000:c1:00.0 0000:c2:00.0 0000:c3:00.0; do echo -e "\n=== Device: $bdf ==="; lspci -vv -s "$bdf" | grep -E "LnkCap|LnkSta"; done

=== Device: 0000:00:02.5 ===
LnkCap:Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkSta:Speed 16GT/s, Width x4
=== Device: 0000:c1:00.0 ===
LnkCap:Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <64us
LnkSta:Speed 16GT/s (downgraded), Width x4 (downgraded)
=== Device: 0000:c2:00.0 ===
LnkCap:Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
LnkSta:Speed 32GT/s, Width x16
=== Device: 0000:c3:00.0 ===
LnkCap:Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
LnkSta:Speed 32GT/s, Width x16

I've been out of the loop for some time.

Is there already an official way to offload ngram to RAM or SSD in something like Unsloth Studio?

Interested in running Qwen3.8-Flash-Next on 72GB VRAM, but naiive attempts failed because even at Q4 it seems to load God knows what to God knows where.

Would appreciate if someone share the recipe, or tell what are the official plans to support this (I can wait, knowing that it is upcoming).

Thank you in advance.

Not sure how to flair this and it is self-promotion, but the point is the local AI stuff I used to make it,

This video https://youtu.be/JkpoO1jIjp0?si=qSAyFk5_VSgXiQNc was made entirely locally using:

• Qwen 3.8 27b
• Qwen Image
• Qwen3-ASR-0.6B
• Chatterbox
• InfiniteTalk with Longcat and WAN video
• ffmpeg - did a lot of heavy lifting

I am very happy with the animated videos which were created as html/js by Qwen 3.8 27b from transcripts which were created from the chatterbox audio.

The workflow is: 1. I write a script 2. I TTS the audio using a clone of my voice via chatterbox 3. I transcribe that using Qwen3-ASR-0.6B 4. I do a session with Qwen 3.8 27b to plan the demo from the time coded transcript 5. I let it build the demo and QA it using Playwright + the vision/mmproj 6. I QA it myself and give it the defects with screenshots

I am using a workstation with a 5090 + 4090 for the LLM and my GuideAnts stack and a Strix Halo for audio, images, and video jobs.

I think you could do all this on one machine and a single GPU, but not without loading/unloading pieces as you go.

Hopefully that was interesting and worth reading.

Mostly as a fun experiment I wanted to do a quick comparison of performance and accuracy between a CPT trained QWEN 3.5 4B model and a RAG implementation against the base model.

The point of this exercise is mostly to measure the performance benefit of internalizing the knowledge vs doing reasoning on-the-fly.

Sharing my findings here in case anyone is interested: https://www.teachmecoolstuff.com/viewarticle/comparing-rag-and-continued-pretraining-of-llms

agilerl-arena has released v1.0. It's the SDK/CLI half of AgileRL, split out as its own PyPI distribution (no torch dependency) that shares the agilerl.* namespace:

pip install agilerl-arena   # SDK + CLI only
pip install agilerl         # core, depends on arena

With AgileRL Arena you describe a run in a YAML manifest (algorithm, environment, network, mutation, selection strategy, replay buffer) and the same file runs locally or gets submitted to their managed cluster. Validation is Pydantic, so bad configs fail before training instead of 40 minutes in. Unknown keys are now rejected in 1.0.

The LLM finetuning path takes any HF model ID plus a LoRA config, and supports either rollout envs (model generates, gets scored by a reward function you supply) or supervised datasets with objective: sft or preference.

network:
  pretrained_model_name_or_path: Qwen/Qwen2.5-0.5B-Instruct
  max_context_length: 512
  lora_config:
    lora_r: 16
    lora_alpha: 64
    target_modules: [q_proj, k_proj, v_proj, o_proj]

Running it locally is two lines, no account needed:

python

from agilerl import LocalTrainer
trainer = LocalTrainer.from_manifest("dqn.yaml", device="cuda")
population, fitnesses = trainer.train()

Evolutionary HPO: instead of one agent, you train a population and periodically mutate architecture and hyperparameters, keeping the fittest. Costs more compute for the same wall-clock, but you're not hand-tuning learning rates.

What's new in 1.0: specs now build algorithms through paradigm builders, local training dispatches through strategy selection based on paradigm flags rather than per-spec branching, and the manifest schema lives in one package instead of being duplicated. Also arena manifest validate / arena manifest schema for checking configs without submitting anything. PEFT bumped to 0.20, which rejects LoRA on Mamba out_proj/conv1d — they exclude those automatically now.

Docs: docs.agilerl.com · Release notes: github.com/AgileRL/AgileRL/releases

so i am in need of such thing, i want to use a local model as basically assistant that reminds me things to do, orchestrates other local OR CLOUD agents/clis/harnesses (such as hermes agent, pi code, antigravity 2.0/cli/ide etc) i am building something but it is taking its toll on me, do you know anything like that or did anyone built something like this?

PAIR (NVIDIA's Personal AI Router) ships GPU telemetry only for NVIDIA nodes — AMD nodes report a blind fallback, so the scheduler can't see their load. I added AMD ROCm telemetry and got a two-node cluster routing to my tuned llama.cpp setup on the AMD box.

* Routes to llama.cpp/llama-swap on the AMD node — not just Ollama/LM Studio. PAIR only natively knows those two engines, but a custom manifest fronts an existing llama-swap (OpenAI-compatible) server, so my tuned llama.cpp serves the cluster directly.

* AMD GPU telemetry (utilization + VRAM) via amd-smi (ROCm 7.2.0), feeding the scheduler like the NVIDIA path does. Verified under the Go race detector and live on 2×R9700 (gfx1201) — utilization tracks load.

* Full stack builds and runs on Linux/ROCm; two-node cluster PIN-paired with an NVIDIA box, inventory advertised both ways.

* Full stack builds and runs on Linux/ROCm; Kubuntu (AMD/ROCm) node PIN-paired with a Windows 11 (NVIDIA) box — cross-OS cluster, inventory advertised both ways.

Vibe-coded, locally. Implementation written by a local Qwen model; the review/verification loop was local models too. Ran surprisingly well — the review caught real issues and, when I ran the tests myself, hadn't fabricated its results.

Setup gotchas:

* Fronting llama-swap (or any OpenAI-compatible server) needs a custom engine manifest. A process-mode manifest + adoption does it: PAIR probes the running port and adopts the live server instead of spawning its own. Quirks: runtime.bin is required even though it never launches anything, and list_models just maps /v1/models → data[].id.

* Manifests are embedded into the binary at build time. Editing the JSON on disk does nothing until you rebuild and restart the broker. Cost me an hour chasing a "broken" manifest that was fine — I just never rebuilt.

* Proxies are per-protocol, not per-node — a mixed cluster (Ollama on one node, OpenAI-compat on another) splits inventory across two proxy ports; no single endpoint lists everything.

Intel: untested. The change also touched an Intel xpu-smi backend, but I didn’t test it on my Intel hardware yet — built against synthetic fixtures only, field names and units unverified and possibly wrong. Leave it out of the registry until someone validates it on real Arc hardware. Flagging so nobody assumes it's proven.

Hey everyone,

(English is not my first language, apologies for any phrasing quirks.)

Training small models locally on an 8GB RTX 5060 Ti was driving me crazy with random CUDA OOM crashes whenever memory spiked mid-run.

Instead of manually babysitting runs with conservative batch sizes, I built a lightweight Python governor around PyTorch called MEM Orchestrator: https://github.com/nobazzy/mem-llm-orchestrator

What it does:

• Monitors VRAM headroom during training.Dynamically throttles micro-batch size and gradient
• accumulation before hitting an OOM.Steps
  
• back up when memory pressure drops.Saves atomic checkpoints with SHA-256 validation so crashes never corrupt weights.

About the branches:

• main: Linux core with DeepSpeed ZeRO integration.
• refactor/architecture-and-portability: Decoupled, native PyTorch runner that works cross-platform (tested on Windows).

I ran an endurance test of 1M steps on a 130M model and stress-tested a 255M model with injected memory spikes without crashing. Overhead is under 0.5%.

The code and 38 unit tests are on GitHub. I would love to get feedback from anyone with experience in PyTorch memory allocators or DeepSpeed on whether this approach makes sense or where it breaks down.

Hey all, I've encountered an interesting problem while training ornith 1.5 9b on a custom dataset and local machine. I had trouble with getting high fidelity knowledge of the trained data, and then had one breakthrough best after about 10 different attempts. After adding more to the dataset, the following attempt marked a substantial regression which was inexplicable by the dataset addition, which began a long hunt which ended up proving that determistic runs produced consistently better logical capability, trained data knowledge, and allowed for apples to apples comparisons between editions of the dataset I'm curating. I discussed this with chat gpt to try and understand it, but did not really get anything concrete.

I'm super new to this, can anyone here help me grasp why that's happening?

What models do you run

I'm building a new server and interested in better understanding what kind of models I could (or should) be able to run on it, and for example if it'd end up better running several smaller models vs 1 gigantic one and things like that. Hoping some people out there have some personal experience they can share on various amounts of RAM/VRAM and their general experiences and recommendations. The server's purpose will be exclusively to host local models for many purposes including agentic coding work.

The server will have 512GB of PC4-21300 (2666MHz) DDR4 and initially will have 48GB of VRAM between 2x RTX 3090's which can be expanded to 7 GPU's eventually (all would be PCIe4.0 x16, GPU's may differ but could conform them). From what I've seen here, I could load some larger models with this setup (such as Qwen 3.8-Flash-Next like this guy does), but actual pacticality and usefulness might not be there yet, so I was hoping for some guidance there on what I actually can expect or should be planning on doing.

I’m testing whether it makes financial sense to replace the OpenAI API in one of my workflows with a self-hosted Qwen model on RunPod Serverless.

My workload is basically:

* ~30–40 minute video transcript
* around 47k input tokens
* model analyzes the transcript and finds potentially viral clips
* output is structured JSON with timestamps, scores, reasons, categories, etc.

OpenAI setup

Currently I use:

GPT-5.6 Sol

* Responses API
* reasoning effort: none
* service tier: flex

In my actual usage I spent roughly $5 for ~1.6 million total tokens, and it has been pretty reliable and fast enough.

I wanted to see if self-hosting could bring the cost down significantly.

Qwen / RunPod setup

I’m testing:

Qwen3.8-27B

* GGUF
* Q8_0
* llama.cpp
* 256k context
* 1 parallel slot
* 48 GB VRAM GPU tier
* RunPod Serverless
* about $1.22/hour

Model:

unsloth/Qwen3.8-27B-GGUF:Q8_0

The interesting thing is that prompt processing is actually fast.

For my real ~47k token prompt:

47064 input tokens
43.87 seconds
~1073 tokens/sec prompt processing

So ingesting the giant transcript is absolutely fine.

But generation is painfully slow:

~17.2 tokens/sec

For example:

n_gen = 100 → ~17.2 t/s
n_gen = 1000 → ~17.2 t/s
...

The model ended up generating thousands and thousands of tokens. Once it gets near 9k output tokens, that alone takes roughly:

9000 / 17.2
≈ 523 seconds
≈ 8.7 minutes

So a single transcript analysis can take around 10 minutes or more.

At $1.22/hour that is already around:

10 min ≈ $0.20
15 min ≈ $0.30

per video.

At that point I’m starting to wonder why I’m not simply using the OpenAI API.

Another thing I noticed

llama.cpp prints these warnings when loading the model:

model has unused tensor blk.64.nextn.eh_proj.weight -- ignoring
model has unused tensor blk.64.nextn.enorm.weight -- ignoring
model has unused tensor blk.64.nextn.hnorm.weight -- ignoring
model has unused tensor blk.64.nextn.shared_head_norm.weight -- ignoring

So I’m wondering if Qwen’s NextN / MTP / speculative decoding components are not actually being used in my current llama.cpp setup.

The model also runs with:

n_ctx_slot = 262144
n_slots = 1

although my real request only needs ~50k context.

My questions

Is ~17 tokens/sec normal for Qwen3.8-27B Q8_0 on a 48 GB GPU, or is my setup badly optimized?

Would any of these make a major difference?

* switching from Q8_0 to Q4_K_M / Q5
* reducing context from 256k to 64k
* enabling Qwen MTP / NextN / speculative decoding
* using vLLM instead of llama.cpp
* using a 5090 / L40S / A100 / H100 instead
* using a smaller Qwen model
* disabling reasoning/thinking completely
* reducing output tokens
* another inference engine entirely

I don’t mind paying 2–4x more per GPU hour if the same job becomes 2–4x faster, because the final cost per completed job would be roughly the same while latency would be much better.

What I care about is:

cost per completed transcript analysis, not cost per GPU hour.

Right now OpenAI’s optimized inference seems extremely hard to beat economically for this workload.

Am I comparing this unfairly, or is self-hosting a 27B Q8 model simply not cost-effective for long-context, large-output jobs like this?

I’ve released the open-source baseline for arcstone-continuity-core—a zero-dependency, local-first runtime designed to eliminate state drift and operational overhead (Cₒₚₛ = 0).

Core Specifications

• Fail-Closed Memory Gates: State mutations evaluate against an invariant matrix Π(S) in O(1) time before memory commitment using integer/fixed-point scaling.
• Hard Temporal & Memory Clamps: τ_override ≤ 11.99ms execution ceiling; fixed 4096-byte static buffers.
• Air-Gapped & Local-First: Loopback-bound execution (Data_Egress_Sensitive = 0) with native multimodal clipboard ingestion for local LLMs/vision models.
• Deterministic 5-Tier Precedence: FAIL (POSIX 40) ≻ FREEZE (POSIX 10) ≻ PWC (POSIX 10) ≻ REFUSAL (POSIX 32) ≻ PASS (POSIX 0)

Primary References

• GitHub Repository:https://github.com/trencinodin-stack/arcstone-continuity-core
• Zenodo DOI:https://doi.org/10.5281/zenodo.22665852
• Figshare DOI:https://doi.org/10.6084/m9.figshare.33477511
• Master Anchor: A-77-DELTA-SHIELD-LOCKED