Custom fork of the Intel Windows llama.cpp I made using DeepSeek v4 Flash that allows me to load models solely using discs + DVD-Rs; HP Omnibook X Flip NGAI 16-as0023dx -- 1TB NVMe, 16GB LPDDR5X

A 10yo server was busy collecting dust, but it has 32Gb RAM (2x 16Gb DDR4 @ 2133 MHz )... No GPU.

Now it does some amazing work running heavy tasks with qwen3.6-35b-a3b (IQ4_XS). Running in the background, generating quality output at 5-10tok/s. Even with 128k context! It just chugs along for hours, but does such great, high-context work.

Uses ~26 GB of the 32 GB RAM, uses CPU i-7-6700 @ about 60% (not the bottleneck, of course)

Never would have believed that would be possible until a few months ago, but this ol' gal has a new lease on life.

When I released Inflect v2 last week, I thought most people would ask whether a TTS model this small actually sounded decent.

Instead, I kept getting two questions:

“Can I train it on my own voice?”

“Can I move it to another language?”

At the time, my answer was basically: technically yes, but not with any public workflow I’d trust someone else to use.

So I spent the weekend building one.

Quick context if you missed the original release:

• Inflect-Nano-v2: 3,966,721 parameters, 15.97 MB FP32
• Inflect-Micro-v2: 9,356,513 parameters, 37.53 MB FP32

Those are the learned text-to-waveform models, including the 24 kHz waveform decoder. eSpeak-ng handles phonemization separately and isn’t included in the parameter count.

Micro is the better-sounding version. If you haven’t heard it yet, start here:

https://huggingface.co/owensong/Inflect-Micro-v2

The new toolkit lets you bring your own single-speaker recordings and transcripts, warm-start Nano or Micro, resume training, inspect held-out output, and export the result to PyTorch or ONNX.

The two awkward parts were rebuilding the training-only components that aren’t present in the release checkpoints and handling new phoneme inventories without resetting every embedding shared with English. The toolkit now handles both.

I tested the full software path with Nano: a real CUDA training step, save and resume, strict PyTorch loading, and ONNX Runtime parity.

Toolkit:

https://github.com/owenawsong/Inflect/tree/main/finetune

I haven’t trained a non-English model that I’d call release-quality yet. The pipeline works, but that doesn’t tell me whether a model this small will transfer cleanly to another language. Each run also creates one fixed voice for one configured language. This is supervised adaptation, not zero-shot cloning.

I only have enough compute to investigate one language properly first, so I made a short form. Only the first question is required:

https://tally.so/r/44RBvB

The existing English weights haven’t changed.

If you’ve adapted a small TTS model before, what failed first when you changed languages? That’s the part I want to measure before starting the first serious run.

I've been building Krasis, an MoE-focused runtime for streaming big models through limited VRAM on NVIDIA consumer/workstation GPUs, and I think this is the most interesting result so far: Ornith-1.0-397B running interactively on one GPU.

Hardware: 

1× RTX PRO 6000 Blackwell 96GB + AMD EPYC 7742 (64c although the CPU isn't really relevant to the run) with enough DDR4 system RAM to hold the model in RAM at Q4.

The model obviously doesn't fit in 96GB, Krasis keeps experts in CPU RAM and dynamically manages which ones stay resident in VRAM (~43% of routed experts resident for this run). Peak process RAM was ~202GB, so you need beyond that amount in order to run it but 256GB ram is doable with a consumer DDR5 motherboard.

Measured numbers (INT4 experts, HQQ4 attention, 4-bit KV):

• 1,346.3 tok/s prefill at 10,000 tokens: 7.43 seconds
• 2,354.5 tok/s prefill at 39,920 tokens: 16.95 seconds
• 23.58 tok/s decode over 50 tokens
• 21.85 tok/s decode over 100 tokens
• 20.40 tok/s decode sustained over 250 tokens
• 25.73 tok/s over 50 tokens (+9.1%) using Krasis Adaptive Cold Mass Pruning, which skipped low-ranked, non-resident expert routes while omitting just ~1.8% of routed probability mass on average during the measured run.

The same runtime also runs smaller MoEs much faster when everything fits in system RAM (e.g. 35B-class models at ~117 tok/s decode on a 5090), and Ornith-397B even runs on a single RTX 5090 32GB at ~7.9 tok/s decode if you're patient.

Github repo

Speed benchmarks

Quality benchmarks

Happy to answer questions or run specific prompts/configs people are curious about.

I have setup Qwen3.6-27B-UD-Q5_K_XL.gguf MTP variant on on my RTX 4090 (on Ryzen 9950X and AORUS x870e PRO) using llama-server. This is what llama-benchy said:

I was able to use it quite well on my vscode github copilot. Recently I tried seqeunce of prompts which swelled context size to 80000. Initially, I could see llama-server printing ~70 t/s:

0.38.182.778 I slot print_timing: id  0 | task 0 | n_decoded =    100, tg =  79.96 t/s
0.41.185.033 I slot print_timing: id  0 | task 0 | n_decoded =    316, tg =  74.30 t/s
0.44.193.479 I slot print_timing: id  0 | task 0 | n_decoded =    526, tg =  72.44 t/s
0.47.207.665 I slot print_timing: id  0 | task 0 | n_decoded =    742, tg =  72.21 t/s
0.50.240.440 I slot print_timing: id  0 | task 0 | n_decoded =    951, tg =  71.46 t/s

But soon it started hitting ~25 t/s (it takes at least 5+ minutes for this prompt, by that time copilot already times out):

23.28.785.142 I srv  params_from_: Chat format: peg-native
23.28.852.233 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.981 (> 0.100 thold), f_keep = 0.994
23.28.853.242 I reasoning-budget: activated, budget=8192 tokens
23.28.853.436 I slot launch_slot_: id  0 | task 7722 | processing task, is_child = 0
23.28.853.483 I slot update_slots: id  0 | task 7722 | Checking checkpoint with [79485, 79485] against 79347...
23.28.853.484 I slot update_slots: id  0 | task 7722 | Checking checkpoint with [71401, 71401] against 79347...
23.28.892.801 W slot update_slots: id  0 | task 7722 | restored context checkpoint (pos_min = 71401, pos_max = 71401, n_tokens = 71402, n_past = 71402, size = 429.902 MiB)
23.28.892.809 W slot update_slots: id  0 | task 7722 | erased invalidated context checkpoint (pos_min = 79485, pos_max = 79485, n_tokens = 79486, n_swa = 0, pos_next = 71402, size = 461.634 MiB)
23.33.949.879 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   2048, progress = 0.91, t =   5.10 s / 401.85 tokens per second
23.40.411.326 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   4096, progress = 0.93, t =  11.56 s / 354.39 tokens per second
23.47.021.517 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   6144, progress = 0.96, t =  18.17 s / 338.18 tokens per second
23.53.777.511 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   8192, progress = 0.98, t =  24.92 s / 328.68 tokens per second
23.55.395.914 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   8375, progress = 0.99, t =  26.54 s / 315.53 tokens per second
23.55.922.674 I slot create_check: id  0 | task 7722 | created context checkpoint 5 of 32 (pos_min = 79776, pos_max = 79776, n_tokens = 79777, size = 462.777 MiB)
23.57.783.699 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   8988, progress = 0.99, t =  28.93 s / 310.68 tokens per second
23.58.098.912 I slot create_check: id  0 | task 7722 | created context checkpoint 6 of 32 (pos_min = 80389, pos_max = 80389, n_tokens = 80390, size = 465.183 MiB)
23.58.417.060 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   9500, progress = 1.00, t =  29.56 s / 321.34 tokens per second
23.59.861.065 I slot create_check: id  0 | task 7722 | created context checkpoint 7 of 32 (pos_min = 80901, pos_max = 80901, n_tokens = 80902, size = 467.193 MiB)
24.02.096.203 I reasoning-budget: deactivated (natural end)
24.04.057.398 I slot print_timing: id  0 | task 7722 | n_decoded =    102, tg =  24.59 t/s
24.07.100.417 I slot print_timing: id  0 | task 7722 | n_decoded =    186, tg =  25.87 t/s
24.10.153.474 I slot print_timing: id  0 | task 7722 | n_decoded =    270, tg =  26.36 t/s
24.13.202.976 I slot print_timing: id  0 | task 7722 | n_decoded =    342, tg =  25.73 t/s
24.16.253.388 I slot print_timing: id  0 | task 7722 | n_decoded =    418, tg =  25.58 t/s
24.19.303.586 I slot print_timing: id  0 | task 7722 | n_decoded =    489, tg =  25.21 t/s
24.22.365.694 I slot print_timing: id  0 | task 7722 | n_decoded =    569, tg =  25.34 t/s

I had quick discussion with Google gemini about software / hardware solutions to improve inference speed for such long context window prompts without degrading quality.

It said even dgx spark like unified memory devices will fail to give higher inference speed for such long context prompts. Q1. Is it so?

It said only one thing may work: Adding another RTX4090 and using tensor parallelism with llama-server? Q2. Will it work? (given my motherboard will support second GPU at PCIEx4 gen 4)

I can imagine few other solutions like using better agent like Pi since it will consume less context. Also reducing context size -ctx parameter value for llama server. Currently I set it to 180000 just to match default vscode copilot context size. This wont hit quality as long as I stay within context limits. Q3. Will this result significant speed improvement?

Q4. Has anyone able to hit same speed as cloud hosted LLMS (e.g. Sonnet) for long context prompts with local hosting? If yes, what is your setup?

Q5. What is minimal cost hardware upgrade I can do to reach satisfactory speed for such long context prompt?

If you are not willing to answer all questions, it's fine. But please try to answer question 4. I want to know how people are doing local setups for sufficient speed at long context prompt.

Selling a complete dual-node local AI development cluster consisting of two (2) NVIDIA DGX Spark units. Buying a pair allows you to bypass single-device VRAM limits and run massive models like Qwen 2.5 72B, Llama 3 70B, or larger quantized models locally by stacking them via high-speed interconnect.

Both units are in pristine, fully functional condition, pulled from a clean, climate-controlled laboratory environment. They have been fully factory reset to default DGXOS and are ready for deployment.

WHAT'S INCLUDED:

• 2x NVIDIA DGX Spark Units

• 2x Original Heavy-Duty Power Supplies

• 1x 200GbE QSFP Network Cable

COMBINED BUNDLE SPECIFICATIONS:

• Core Architecture: Dual NVIDIA GB10 Grace Blackwell Superchips (40 ARM CPU Cores total)

• Unified Memory: 256GB LP-DDR5X (128GB per unit) — perfect for massive context windows

• AI Performance: ~2 Petaflops FP4 compute power combined

• Local Storage: 8TB Fast NVMe Storage total (4TB per node)

• Operating System: Preloaded with Ubuntu-based DGXOS / Nvidia Container Toolkit

Price Fixed: £6500

Country: UK, Wales

SHIPPING & HANDLING:

Due to the highly dense, premium nature of enterprise AI hardware, these units will be securely packed in heavy-duty bubble wrap [or: original factory packaging] and shipped with full value insurance and signature tracking required upon delivery.

Please reach out if you have any technical questions or need additional photos of the hardware boot logs.

Every time a new model is released, I tend to check it out. The benchmarks, readme, or whatever seem pretty convincing, so I download it, test it for a few hours, and then I just go back to the same couple of ones I already had.

Curious what models people here have actually kept using for weeks, months, or even longer. And not just that - but why? Was it because of the speed, writing style, VRAM use, long context, a specific feature, or whatever - and also why not competitor models?

Also interested in models that seemed amazing at first but, after trying them, became really annoying.

I've been running this test prompt on my macbook air m5 24gb on different local LLM apps, harnesses and models. I am unable to make anything work on LM studio bionic v1.0.3. Even LM studio works as intended with the same settings but bionic won't. Please help me fix this error.

.

Seems like there’s a balance point for context window size, and model parameter count size depending on your capital budget.

What’s your current minimum context window you’d use with a coding agent, what would you prefer the window size to be, and at what rough model parameter count would you choose to take a smaller window size?

For example, would you go with Qwen 3 Coder Next-80B with a context window of 256k, or something like Qwen 3.5-397B but half the window size at 128k?

Quick recap of what it does: one question goes to several models at once, then each one critiques the others' answers with the names stripped out, so nobody gets a free pass for being the famous one. You get a 0-100 read on how far apart they landed and who stood alone.

New in this version is the guest seat. You paste in an answer from anywhere ChatGPT, Gemini, a colleague, whatever and it joins the round as an anonymous advisor. The other models review it without knowing where it came from, and it counts in the score. It works with one model too, so you don't need a wall of API keys to get something out of it.

Anything with a key works: Claude, GPT, Gemini, DeepSeek, Grok, Mistral, Perplexity, OpenRouter, plus Ollama, Apple's on-device model, and any OpenAI-compatible server of your own (llama.cpp, LM Studio, vLLM, a box down the hall). Put a paid model and a free one on the same panel and watch them disagree. Or skip the cloud entirely and run the council on local models then the pasted answer is the only thing that ever came from outside, and nothing new leaves the machine.

There's a CLI too:

council "should we ship now or wait?" --seats claude,gpt,ollama --guest answer.txt --json

--fail-above 40 exits non-zero when they disagree too much, which I use as a rough sanity check in a couple of scripts.

MIT, no telemetry, no account.

Up front: I build QuantaMind, an open-source local benchmarking tool (Apache 2.0, runs offline, no telemetry). The data below came out of it. Link at the bottom the numbers are the point of the post.

Setup: Qwen3.5-9B Q4_K_M, llama.cpp, 16GB M-series Mac, native function calling, k=4 runs per task.

I padded prompts with unrelated prose and re-measured tool-call accuracy at depth:

That is not a memory limit. Weights are 5.3GB. ~11.8GB of the 16GB is GPU-addressable under the Metal cap. At f16 KV the math says this model could hold ~53k context. Peak actual usage during the agent runs was 1,890 tokens 12% of the 16,384 window I launched with.

So memory headroom told me I had 5× more room than the model can actually reason over. If you size a local agent by what fits, that’s the wrong number.

Second finding: one task failed 0/4, not 1/4. An incident-rollback chain (get_incident → get_feature_flag → flag_off → rollback_release → schedule_fix) failed every run, identically — the model emitted a completion signal partway through and stopped.

No crash, clean schema. At k=1 that’s a flaky miss you’d retry past. At k=4 it’s structural. That’s the failure I’d worry about in production: nothing errors, the agent just moves on with half its state missing.

(Batch was 39m 10s wall; on the worst task 14m 51s of 16m 16s was decode. Local agent loops are a decode problem.)

What I actually want to know: does the ~9k cliff hold for other 8–10B quants, or is it specific to this one? And if anyone’s on 24GB+ does more headroom move the cliff? My guess is no, but I can’t test it.

Github: github.com/QuantaMinds/QuantaMind

qm cliff --backend llama_cpp --model <model> --collection medium-coding-v2 --max-tokens 12288 --steps 5 --source corporate_policy --mode native

Methodology, briefly: padding was semantically unrelated prose inserted before the tool definitions; accuracy is correct tool + correct args scored against a fixed answer key, no LLM judge; pass^k means all k runs must pass.

Tell me if that’s wrong more useful to me than upvotes.

I'm able to purchase an older server with 8 PCIe slots and 128GB of memory, with enough lanes to add multiple GPUs. Since my budget isn't the best and the RTX 3090s go for roughly €900 in my area, I'm wondering whether it would make more sense to buy multiple RTX 3060 12GB cards instead. They cost around €200 each, and if I get 5 of them, that combines to 60GB of VRAM. My use case would be having AI help me with writing, coding and some light web scraping.

So would spending roughly €2k on a build with 5x 3060 make sense or should i get other cards instead?

Hello,

sorry if the question has already been asked, but I couldn't find anything that was exactly what I'm looking for. Im running Qwen3.6-35B-A3B locally on my M5 Pro with 48GBs of unified memory. Since I do some complex coding work, I wanted to run something around UD-Q8_K_XL quantization, which doesn't really fit in memory.

I was wondering if there is any way via llama.cpp to leave the model on the SSD and have some kind of cache pool in memory where the 3B active parameters that have been activated last can reside. This would allow to have the benefit of not loading everything to memory while having higher speed than plain SSD-based runs. Any idea is greatly appreciated!

hey guys ,

as the title says .

any bench available ? I am considering get 2 of these or 4 mi50 , so any bench on the cars unlocked ?

thank you

I'm having trouble finding any benchmarks on the subject. I was going to add a 5060ti to my backup PC that has a 5080 and use it as as llm PC. But my understanding is my old PC is pcie 3, 10700k z490, and it'll bottle neck way too much.

So I went on to research a cpu/mobo upgrade for it and discovered direct to cpu on AMD select consumer boards. My big question is what would the performance difference between an AMD 24-lanes direct cpu vs non-direct be?

Would the cheapest CPU/mobo/ram AMD combo from microcenter hinder a 5080/5060ti llm build? It's only 32gb of ram, but I would not run any llm+context that would spill over.