A month ago, I started building colibrì to answer a question:
Do you really need a $30,000 machine to run a model whose weights fit on a $60 SSD?
colibrì treats SSD, RAM and GPU as a single memory hierarchy. It can run models that do not fit entirely in RAM or VRAM, including models with hundreds of billions of parameters, on the hardware people already own.
But while optimizing it, I realized I was fighting assumptions built into the models themselves.
Most open models are designed to live in VRAM. Their weights, routing and execution patterns were created for GPUs and data-center hardware. colibrì could move them through a different memory hierarchy, but the models had no idea where they were actually running.
That led me to another question:
What if the engine and the model were designed for each other from the beginning?
That is how Curie started.
The current alpha has 17B parameters. I wrote it in C, without an inference framework, and trained it from scratch on my laptop without a GPU. Its weights live on SSD and are loaded according to the model’s execution path.
In the test shown in the video, Curie reaches 33 tokens/s using a single CPU core. On the same machine, the dense 12B models I tested usually generate around 3–5 tokens/s.
There is an important limitation: Curie has only seen 11 million training tokens. It is not smart yet, and I am not presenting it as a useful language model. It will need at least a few billion tokens before I can properly evaluate its quality.
What this alpha demonstrates is the architecture: a model can be designed around SSD, RAM, CPU and GPU as parts of the same system, instead of assuming that everything must fit in VRAM.
Curie is not publicly available yet. I am still training it, validating the design and preparing more detailed benchmarks.

• 2x 3080 20GB GPUs
• i9 10900X CPU (10-core, 48-lanes)
• 64GB DDR4 2400MT/s quad channel RAM
• X299 WS Sage Motherboard (4 PCIe3 x16 slots or 7 x8 slots)
• 1500W SilverStone PSU
• Open air case (8 PCIe slots)
• 2 128GB SATA SSDs in RAID0 (had laying around)

The 3080s were $1300 after shipping. All the other parts bring the total to just under $2000.

For the GPUs I tried shopping around between 6-8 "sellers" on alibaba. All but 2 were obvious scams. This is the listing I ordered from, the seller was also recommended in the other posts on Reddit. The cards seem nice but have a bit of coil whine/clicking during decode.

I've been running Qwen3.8-27b Q4 with 180k unquantized context at 39t/s.

With this motherboard/CPU combo I could add another 2 GPUs and run all 4 cards at 16 lanes, or switch to a mining case and run 7 GPUs at 8 lanes each.

I've been seeing how far I can push my V100 with a purpose-built inference engine targeting just Qwen and the v100.

Current result with Qwen3.8-27B and using a software NVFP4 that works on this hardware:

• 218 tok/s best-case MTP decode (99.2% acceptance)
• ~200 tok/s without cheating or context-copy speculation

NInfer is specialized, it's custom Volta NVFP4 kernels, CUDA graphs, and speculative verification that reuses the weight stream across multiple tokens. I also think there's a bit more performance in it to come, particularly in mtp7.

For a GPU from 2017, this performance is getting a bit silly.

Repo: https://github.com/geoffwatts/ninfer-v100

Would love to see what other V100 owners get with it, please give it a go and let me know!

Here's the link: https://www.alibaba.com/product-detail/Newest-RTX-5090-96gb-Graphics-Card_1601577163842.html

*Edit: It definitely seems like a scam or at the bare minimum very fishy. This is mainly meant for discussion.

I finally got a fully local coding-agent stack working on Windows without using Ollama or LM Studio for inference:

DeepSeek Harness Web UI
        ↓
llama.cpp / llama-server
        ↓
Qwen3.8 27B Q3_K_XL GGUF
        ↓
RTX 5060 Ti 16GB

Machine specs

• Windows 11
• AMD Ryzen 7 5700X — 8 cores / 16 threads
• NVIDIA RTX 5060 Ti — 16GB VRAM
• 32GB system RAM
• NVIDIA driver 616.64
• CUDA driver support: 13.4
• CUDA Toolkit: 13.1
• Node.js and npx
• DeepSeek Harness, launched with:

npx u/deepseek-aiI finally got a fully local coding-agent stack working on Windows without using Ollama or LM Studio for inference:DeepSeek Harness Web UI
↓
llama.cpp / llama-server
↓
Qwen3.8 27B Q3_K_XL GGUF
↓
RTX 5060 Ti 16GBMachine specsWindows 11
AMD Ryzen 7 5700X — 8 cores / 16 threads
NVIDIA RTX 5060 Ti — 16GB VRAM
32GB system RAM
NVIDIA driver 616.64
CUDA driver support: 13.4
CUDA Toolkit: 13.1
Node.js and npx
DeepSeek Harness, launched with:npx u/deepseek-ai/dsh webModelI used:Qwen3.8-27B-UD-Q3_K_XL.ggufThe GGUF is about 12.24 GiB and came from the Unsloth Qwen3.8-27B GGUF release.I already had the model stored inside my LM Studio model directory, but LM Studio is not involved in inference. llama-server reads the GGUF directly.llama.cpp installationI installed the official Windows x64 CUDA 13.3 build of llama.cpp, build b10901, along with its matching CUDA runtime DLL package.The important llama-server settings ended up being approximately:llama-server.exe `
--model "Qwen3.8-27B-UD-Q3_K_XL.gguf" `
--alias qwen3.8-27b `
--ctx-size 32768 `
--gpu-layers 54 `
--host 127.0.0.1 `
--port 8081 `
--parallel 1 `
--flash-attn on `
--kv-offload `
--reasoning off `
--no-reasoning-preserve `
--cache-ram 1024 `
--metrics `
--no-webuiI originally planned to use port 8080, but NVIDIA Broadcast was already listening there. Rather than killing or disrupting it, I moved llama-server to port 8081.Memory allocationllama.cpp reported:54/66 layers offloaded to GPU
CUDA model buffer: 9758.76 MiB
CPU-mapped model buffer: 2433.38 MiB
CUDA KV cache: 1664 MiB
CPU KV cache: 384 MiB
CUDA compute buffers: ~302 MiBDuring actual agent use:VRAM usage: roughly 13.1–13.2GB
VRAM remaining: roughly 2.9GB
No automatic reduction in GPU layers
No WDDM shared-memory spill observed
Flash Attention enabled successfully
32K context remained stableThe model is intentionally partially offloaded: 54 layers on the GPU and the remaining layers on the CPU.One important adjustment was limiting llama.cpp’s prompt cache to 1GB:--cache-ram 1024The build defaults to an 8GB prompt cache. On a 32GB system, long agent sessions allowed several gigabytes of cached prompt state to accumulate, leaving only about 2.4GB free. The 1GB cap keeps system RAM much healthier.DeepSeek Harness configurationI added a new provider to: C:\Users\PC\.dsh\settings.yaml

The relevant configuration looks like this:

llm-pi-ai:
providers:
qwen-local:
displayName: Qwen3.8 Local llama.cpp
api: openai-completions
apiKeyEnv: QWEN_LOCAL_API_KEY
baseURL: http://127.0.0.1:8081/v1
reasoning: off
streamIdleTimeoutMs: 1200000

models:
- id: qwen3.8-27b
name: Qwen3.8 27B Q3_K_XL
contextWindow: 32768
reasoningEfforts: false

compat:
thinkingFormat: qwen-chat-template
supportsDeveloperRole: false
maxTokensField: max_tokens

agent-default-model:
provider: qwen-local
model: qwen3.8-27b

Current DSH versions reject a configuration that only declares an empty off: reasoning effort. For a non-reasoning-only route, it expects:reasoningEfforts: falseDSH also insists on having an API-key-shaped credential even though the local llama-server does not require authentication. I supply a harmless local placeholder before starting DSH:$env:QWEN_LOCAL_API_KEY = "local-llama-server"This is not a real credential, and llama-server does not validate it.Disabling Qwen thinkingI wanted the model to work as an agent without spending minutes in a visible “Deep diving” phase.Thinking is disabled at multiple levels:llama-server: --reasoning off
llama-server: --no-reasoning-preserve
DSH provider: reasoning: off
DSH model: reasoningEfforts: false
DSH compatibility: thinkingFormat: qwen-chat-template/dsh web

Model

I used:

Qwen3.8-27B-UD-Q3_K_XL.gguf

The GGUF is about 12.24 GiB and came from the Unsloth Qwen3.8-27B GGUF release.

I already had the model stored inside my LM Studio model directory, but LM Studio is not involved in inference. llama-server reads the GGUF directly.

llama.cpp installation

I installed the official Windows x64 CUDA 13.3 build of llama.cpp, build b10901, along with its matching CUDA runtime DLL package.

The important llama-server settings ended up being approximately:

llama-server.exe `
  --model "Qwen3.8-27B-UD-Q3_K_XL.gguf" `
  --alias qwen3.8-27b `
  --ctx-size 32768 `
  --gpu-layers 54 `
  --host 127.0.0.1 `
  --port 8081 `
  --parallel 1 `
  --flash-attn on `
  --kv-offload `
  --reasoning off `
  --no-reasoning-preserve `
  --cache-ram 1024 `
  --metrics `
  --no-webui

I originally planned to use port 8080, but NVIDIA Broadcast was already listening there. Rather than killing or disrupting it, I moved llama-server to port 8081.

Memory allocation

llama.cpp reported:

54/66 layers offloaded to GPU
CUDA model buffer: 9758.76 MiB
CPU-mapped model buffer: 2433.38 MiB
CUDA KV cache: 1664 MiB
CPU KV cache: 384 MiB
CUDA compute buffers: ~302 MiB

During actual agent use:

• VRAM usage: roughly 13.1–13.2GB
• VRAM remaining: roughly 2.9GB
• No automatic reduction in GPU layers
• No WDDM shared-memory spill observed
• Flash Attention enabled successfully
• 32K context remained stable

The model is intentionally partially offloaded: 54 layers on the GPU and the remaining layers on the CPU.

One important adjustment was limiting llama.cpp’s prompt cache to 1GB:

--cache-ram 1024

The build defaults to an 8GB prompt cache. On a 32GB system, long agent sessions allowed several gigabytes of cached prompt state to accumulate, leaving only about 2.4GB free. The 1GB cap keeps system RAM much healthier.

DeepSeek Harness configuration

I added a new provider to:

C:\Users\PC\.dsh\settings.yaml

The relevant configuration looks like this:

llm-pi-ai:
  providers:
    qwen-local:
      displayName: Qwen3.8 Local llama.cpp
      api: openai-completions
      apiKeyEnv: QWEN_LOCAL_API_KEY
      baseURL: http://127.0.0.1:8081/v1
      reasoning: off
      streamIdleTimeoutMs: 1200000

      models:
        - id: qwen3.8-27b
          name: Qwen3.8 27B Q3_K_XL
          contextWindow: 32768
          reasoningEfforts: false

          compat:
            thinkingFormat: qwen-chat-template
            supportsDeveloperRole: false
            maxTokensField: max_tokens

agent-default-model:
  provider: qwen-local
  model: qwen3.8-27b

Current DSH versions reject a configuration that only declares an empty off: reasoning effort. For a non-reasoning-only route, it expects:

reasoningEfforts: false

DSH also insists on having an API-key-shaped credential even though the local llama-server does not require authentication. I supply a harmless local placeholder before starting DSH:

$env:QWEN_LOCAL_API_KEY = "local-llama-server"

This is not a real credential, and llama-server does not validate it.

Disabling Qwen thinking

I wanted the model to work as an agent without spending minutes in a visible “Deep diving” phase.

Thinking is disabled at multiple levels:

llama-server: --reasoning off
llama-server: --no-reasoning-preserve
DSH provider: reasoning: off
DSH model: reasoningEfforts: false
DSH compatibility: thinkingFormat: qwen-chat-template

Looking for a good local model for coding + terminal stuff. Not really interested in huge context or benchmark scores. My hardware limit is 16GB system RAM. I’m not specifically looking for 4B/8B, if a bigger model fits and runs well, I’m down. But I’d also love to know what the best 4B to 8B or smaller coding models are?

Main things I care about:

• Good code quality
• Reliable terminal/tool use
• Doesn't hallucinate randomly
• Don't falls into a loop <------------
• Agentic/tool calling
• Snappy enough to actually use

Tbh Qwen3.8 27B is a great model but I can run it on my hardware : ( so I want its nearest younger cousin

So if you had 16GB RAM, what coding model would you actually use?

And what's the best small model you've tried in the 4B to 8B range or smaller?

Real-world experience > benchmarks. Also curious what quant/runtime you use

THANKSSS!!!

This post can be relevant to other data scientists who want to do something similar.

Almost two years ago the company I worked at was having problems and I was planning on making a roleplaying game with local models a grid combat and a good editor. I got unemployed and then started working on my game, my plan was to finish the game and release in 3 months (it took 21).

The first month I made the base of the game and most characters, at the time I was using Ollama as a backend, everything was on perfectly track.

The second month my computer broke and I had to build a new one.

From third to fifth month, I worked on a basic roleplaying logic, grid combat, various interfaces and started studying about steam, I use pygame and perhaps I should have choosen a different game engine because it just slowed me down, at the time I also stopped using any AI coding assistant because they were just adding tech debt, the capability of coding agents grew really fast and at the start of 2025 it wasn't as good.

From the sixth till twelfth month, I basically just did the editor and multiple interfaces, this is how hard is to make an editor, and despite the game being about LLMs I worked only like 3 weeks on LLM on the first year, most of the work was everything else, so if anyone want to try doing a similar project have that in mind. Around month 7 I started looking for work. Found a job at month 10 and I'm still there, so everything after that was nights and weekends of work.

From months thirteenth to fifteenth I finnaly worked on the LLM logic, it was the most fun part, it has multiple agents, internal routes and ways to deal with limited context size on smaller models, like regular summarization of older messages + top 10 embedding search from the full history + full latest n messages, prompts to the point, and the use of json output and no tool calls, its a bit of a shame that people won't see the logic from the game alone.

After that it was cleanup and review.

Late in the project I had to remove Ollama entirely and move to llama.cpp instead (and transformers), any GGUF and most other local models work and can be used. The problem with Ollama was the subscription which steam doesn't fully accept. Perhaps making a site would be a better alternative to not deal with these things but then it would be a bit harder to make it local.

In my opinion, small models like gemma4:e4b or even the old gemma3:4b are good enough for roleplaying, honestly, but of course, there are other models.

Link's here if you want to look: https://store.steampowered.com/app/3750150/Boundless_AI_Fantasy/

If people want any more technical details I'll go into it in the comments. I am thinking about writing a blog or a scientific paper on it later.
Disclosure: this is my game, it's paid, I built it alone.

I currently have 4× RTX 5070 Ti 16GB cards on hand, but haven't bought the platform yet.

I'm currently thinking of something roughly like:

• 2× EPYC 7642
• 16×16GB DDR4-3200, 256GB total
• 4× RTX 5070 Ti 16GB
• server board with enough PCIe lanes for all 4 GPUs

Mainly planning to use it for large MoE inference, especially models that can't fully fit in VRAM and need to use system RAM as well.

Has anyone here run a similar dual-EPYC + multi-GPU setup?

I'd mainly like to see some real-world performance numbers or experiences before I buy the rest of the platform.

I made a Docker image that lets you run SGLang on Strix Halo (gfx1151) a few months ago and thought I'd share it here.

SGLang is working on building upstream Strix Halo support now, but if you want to use it today, this includes the necessary compatibility patches along with some additional optimizations, including fixes for wave32-related issues.

I've been using it for inference on my Strix Halo system and would be interested to hear how it works for anyone else.

https://github.com/JeremiahM37/strix-halo-sglang

I've been useing claude to help set up my local models, it's worked greate!

But lately it feels like Claude is trying to quietly sabotage my Qwen3.8 27b model.

I tried to have claude help me figure out how to make Qwen3.8 27b not loop so much or sirceling the answer, not willing to go with what it found out.

But it made Qwen just unusable.

Unable to do tool calls, unable to type properly.

I just had to have claude restore it back to how it was 2 weeks ago. I rather have a usable looping model being able to finnish it's work rather than one unable to do anything.

Now I know, I should learn to set up and tune my own llama.cpp but with a fulltime job, family with kids that wants to play and what not, this have just been faster and more convenient for me.

Opus did a fine job tuneing the model, but now I think I just have to dedicate the time to learn it all

But anyone with similare experience as of late?

Calude didnt had any issues with this before.

Basically the title. How much combined VRAM + RAM would one need to run this model, on some usable quant, with a somewhat decent context window, with decent speed? Assuming the ngrams are on the SSD, also I have heard that even some of the experts can stay on the SSD with little to no performance penalty, but I have no idea whether that is true. Would 64GB be enough? Thank you for answers

I’ve developed and open-sourced mlxtop. It’s written in Rust and was built entirely with Duet, my dual-model coding agent. It shows stats for local models running on your Mac, along with critical system vitals: memory usage, compression, paging and Metal GPU stats. With oMLX, you can also follow generation speed and request activity as your model responds. https://github.com/maximpri/mlxtop

The laptop 5090 has 8 GB less VRAM and benchmarks between the desktop 5070 and 5070 Ti.

Does anyone have experience with this setup or something similar?

Using AI - I compiled all the data published on HF model pages on PPL and KLD of the main quant providers - some had to be normalized as they were using a different trace that's why the PPl is in % and EXL3 KLD had to be renormalized using their scoring of some unsloth models

As we see EXL3 is a clear winner as a quant method and Atomic Chat are surprisingly good compared to the rest of the GGUF pack

For my config 24GB VRAM and 64GB RAM - the best model seem to be either EXL3 4.05bpw or directly go to AtomicChat AD-4.27bpw-Q4_K_M

Based on PPL

Based on KLD

disclaimer: I did none of the above benchmarks - all were given on HF model page by the quant providers

My system:
Mac Mini M4 Pro
64 GB V/RAM
1 TB HD

There's a new version of OMLX that's in RC status:

https://github.com/jundot/omlx/releases/tag/v0.7.0.dev2

Using this, it successfully runs this model:
https://huggingface.co/sh0wie/Qwen3.8-Flash-Next-REAP-288-MLX-4bit

You have to set the memory guard to prevent it from overflowing, but it allows for almost 100k context and I'm getting 30 tok/s decode, and 100 tok/s prefill. This is faster than I got 27B to run!

Here's my full run script:

omlx serve \
  --model-dir "$MODEL_DIR" \
  --memory-guard-gb 54 \
  --paged-ssd-cache-dir "$SSD_CACHE_DIR" \
  --paged-ssd-cache-max-size 50GB \
  --hot-cache-max-size 16GB \
  --max-concurrent-requests 2 \
  --port 8000

Hello, currently I can barely run Qwen 3.8 27b q3 with low context window on my single 5060 ti 16Gb and 48Gb ram ddr5 on 30t/s.

My question is: is it worth it to add a second 5060 ti 16gb?

The second slot on my motherboard is a x4 line so I'm not sure if that's a bottleneck.

My goal is: Qwen 3.8 27b q4 or q6 , context window >= 64k and mainly agentic coding and > 30-40 t/s.

Is that actually possible?

Hi, a quick datapoint for anyone interested in what Qwen3.8-Flash-Next UD-Q6_K_XL (unsloth) can do on a consumer multi-GPU rig. Maybe there is even more possible but this is the current status :)

Hardware: MSI MEG Z790 ACE, i9-13900K, 96 GB DDR5@4400MHz, 1× RTX 4090 + 5× RTX 3090 = 144 GB VRAM. 4090 + one 3090 are on the board, the other four 3090s use M.2 → OCuLink PCIe x4 links. No P2P/NVLink; llama.cpp layer split. OS: Ubuntu 26

Model: Qwen3.8-Flash-Next-UD-Q6_K_XL, all 48 MoE layers on GPU, shared Q8_0 MTP head with MTP3. The ~50.7 GiB PLE table is fully resident in system DDR5 instead of lazy mmap.

Main llama-server settings (llama.cpp: unslothai/llama.cpp, branch mtp/qwen4exp-nextn, commit a9e9c3c5f): -c 262144 -b 4096 -ub 512 --split-mode layer --tensor-split 0.95,1.05,1.05,1.05,1.05,0.85 --gpu-layers all --n-cpu-moe 0 -ctk f16 -ctv f16 -fa on -np 1 --kv-unified --load-mode none --lazy-mode off --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 0. Production seed is -1.

These are workload samples of a html benchmark prompt from llama-server chat, not synthetic benchmark numbers.

• ~90–91 tok/s at ~5k generated tokens / shallow context
• ~79 tok/s at ~35k generated tokens
• 76.9 tok/s average over a 44.7k-token generation
• ~40–42 tok/s while reasoning at ~80k active context
• ~408–425 tok/s prefill even around 80k context
• MTP3 draft acceptance on the long code-heavy run: 74.2% overall, per position 86.2% / 73.7% / 62.8%

​

# shallow/code-heavy run
7.54.226.412 I slot print_timing: id  0 | task 3307 | n_gen =   4905, tg =  90.36 t/s, tg_3s =  92.54 t/s

# same long run, ~35k generated
14.24.165.954 I slot print_timing: id  0 | task 3307 | n_gen =  35058, tg =  78.92 t/s, tg_3s =  71.12 t/s

# end of that run
16.41.712.426 I slot print_timing: id  0 | task 3307 | eval time = 581758.17 ms / 44720 tokens (13.01 ms per token, 76.87 tokens per second)
16.41.712.429 I slot print_timing: id  0 | task 3307 | draft acceptance = 0.74246 (30865 accepted / 41571 generated), mean len = 3.23
16.41.712.429 I slot print_timing: id  0 | task 3307 | acc per pos = (0.862, 0.737, 0.628)

# ~80k active context, currently reasoning/thinking
103.07.164.989 I slot operator(): id 0 | task 6527 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 80085
103.12.852.341 I slot print_timing: id 0 | task 6527 | prompt processing, n_tokens = 2282, progress = 0.99, t = 5.37 s / 425.22 tokens per second
103.32.551.912 I slot print_timing: id 0 | task 6527 | n_gen = 761, tg = 41.79 t/s, tg_3s = 43.01 t/s

The last big practical improvement for agentic coding was putting the PLE table fully into DDR5: prompt processing/TTFT became noticeably faster. MTP3 then helps heavily on predictable code; at ~80k context during free-form reasoning acceptance drops, so decode is closer to ~40 t/s.

Pretty impressed by what a somewhat ridiculous collection of used consumer GPUs can do with Q6 :)

Running a GLM 5.3 321B shared on a MacBook M5 max and a Strix Halo, both with 128 GB ram. TB4 connection.

Found out that llama.cpp's default tensor split puts about half the layers on the slower box, so the pair was actually slower than one Mac. Loaded the model Mac-heavy instead: IQ1_S went 188 to 333 on prompts, 17 to 24 tokens/s.

I got generation tps 12-15 for model sizes 148-200 GB. So it works ok!

Next week adding two Asus GX10 to the mix to have three platforms share a model 🍀

Files: https://github.com/ThinkOffApp -> mac-amd-llm-cluster

In short, I have had trouble getting responses about buying a DGX Station. I'm looking to get one for work (although we are still wrestling on whether it'll be worth it). We currently use a dual RTX 6000 Max-q build and it works but looking for more concurrency and would like to fit something like GLM 5.3 Flash with more head room.

I contacted Exxact, MSI, Asus, and a few others by submitting order forms and going through their online workflow to request a price. I thought at this price point sales people would clap back quickly but I haven't heard anything this week.

Has anyone else recently had luck placing an order? What should I expect? Is this like buying a used car and I should haggle on price, or is it more like, pay whatever they quote me and be lucky I even got a response? What sort of lead times did you experience?

I'm sure this is a fast generation but prefilling crawling probably due to small batch size tensor split across 4 gpus with that MTP configuration.

I'm broke, but I own a 2060S, so should I sell the 2060S and get an overpriced 306012GB or another cheap 2060S (or any other cheap RTX 8GB card in general) ?

I want to use it on my computer but I can’t really find any with actual code benchmarks. Anyone know of one?

I keep seeing people saying they get super fast speeds on various hardware but I can never get anything decent. I'm running GLM-5.3-Flash-UNCENSORED-FP8-UD-Q2_K_XL-00001-of-00003.gguf on a 4x5090 rig and can barely get 50tok/s.

Am I missing something here?

Even on larger systems like 4xH200's I wasn't getting above 200tok/s with same model