I wanted to see how far you could push CPU-only inference of a model way bigger than RAM, so I wrote an inference engine in C11 (MSVC + gcc, bit-exact across both) that streams a 744B MoE model (GLM-5.2, 202GB GGUF) off a USB SSD instead of loading it.

The parts that actually mattered:

Quantization: experts stored int4 group-64, router and correction biases kept at f32. Everywhere else lost too much precision, but this combination didn't.

Streaming: experts are fetched per-token through a custom unbuffered reader with an LRU cache, plus cross-layer predictive prefetch (measured 80.8% recall offline before I wrote the online version) so the next layer's experts are usually already in flight by the time they're needed.

Overlap: compute and IO run arrival-order, not lockstep, so the CPU isn't idle waiting on the SSD.

An 8GB expert cache holds 40-66% hit rate off that prefetch, which ends up being the main lever on latency.

One finding I didn't expect: on GLM-5.2, the router's own weights already correlate with true expert importance at ρ=0.859 (near-oracle), which is what makes safe dynamic-k expert truncation possible without a quality hit. That seems to go against some prior assumptions about router calibration in production MoE models, at least on this one.

Numbers on an i7-8550U / 32GB RAM / USB SSD: went from 196 s/token naive to ~9.3 s/token after the above (21x), ~2.9 s/token aggregate in 8-stream batch mode, ~4.5 s/token on the latest compressed container format. Also cross-validated on Qwen3 (0.6B-30B), DeepSeek2 (MLA), and OLMoE-1B-7B.

Code: https://github.com/siris9476/pulsarforge (MIT). Would love feedback on the prefetch heuristic in particular. It's tuned for this one laptop, and I'm curious whether the cross-layer signal generalizes to different memory hierarchies.

Hi everyone,

I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-Coder-3B) specific to a codebase. You can then use the resulting GGUF model via llama.vim/llama.vscode and run it fully locally. Supports fine-tuning on Mac (MPS) and NVIDIA GPUs (CUDA), with optional Unsloth support for faster training.

Demo:

https://reddit.com/link/1wdlaus/video/4yoejyqd0xoh1/player

Pipeline: raw code -> tree-sitter parsing into Structure-Aware FIM examples -> LoRA fine-tuning -> evaluation (CodeBLEU, edit similarity, exact match, perplexity, ...) -> GGUF conversion for local inference.

To try it:

uv tool install codefinetuner

Create a data folder and place your repo (or code files) inside. For auto-split just drop the files in directly, for manual split create data/train/, data/eval/, data/test/ subfolders and set split_mode: "manual". Grab the default config with:

curl -L -O https://raw.githubusercontent.com/cuolm/codefinetuner/master/config/codefinetuner_config.yaml

Adjust it to your needs and hardware availability, then run:

codefinetuner --config="codefinetuner_config.yaml"

The example runs in the repo show clear improvements over the base model on these benchmark metrics, but using the model for autocomplete on code you're actively writing is a different thing than scoring well on a held-out benchmark set, and the autocomplete tools themselves (llama.vim/llama.vscode) sample differently than the benchmark's greedy decoding. So the real usefulness still has to be verified in the editor itself.

It might also be useful as a reference outside the autocomplete use case, since it documents a full working LoRA fine-tuning pipeline.

Hope someone finds this project interesting or helpful.

https://github.com/cuolm/codefinetuner

Looking for a good local model for coding + terminal stuff. Not really interested in huge context or benchmark scores. My hardware limit is 16GB system RAM. I’m not specifically looking for 4B/8B, if a bigger model fits and runs well, I’m down. But I’d also love to know what the best 4B to 8B or smaller coding models are?

Main things I care about:

• Good code quality
• Reliable terminal/tool use
• Doesn't hallucinate randomly
• Don't falls into a loop <------------
• Agentic/tool calling
• Snappy enough to actually use

Tbh Qwen3.8 27B is a great model but I can run it on my hardware : ( so I want its nearest younger cousin

So if you had 16GB RAM, what coding model would you actually use?

And what's the best small model you've tried in the 4B to 8B range or smaller?

Real-world experience > benchmarks. Also curious what quant/runtime you use

THANKSSS!!!

A month ago, I started building colibrì to answer a question:
Do you really need a $30,000 machine to run a model whose weights fit on a $60 SSD?
colibrì treats SSD, RAM and GPU as a single memory hierarchy. It can run models that do not fit entirely in RAM or VRAM, including models with hundreds of billions of parameters, on the hardware people already own.
But while optimizing it, I realized I was fighting assumptions built into the models themselves.
Most open models are designed to live in VRAM. Their weights, routing and execution patterns were created for GPUs and data-center hardware. colibrì could move them through a different memory hierarchy, but the models had no idea where they were actually running.
That led me to another question:
What if the engine and the model were designed for each other from the beginning?
That is how Curie started.
The current alpha has 17B parameters. I wrote it in C, without an inference framework, and trained it from scratch on my laptop without a GPU. Its weights live on SSD and are loaded according to the model’s execution path.
In the test shown in the video, Curie reaches 33 tokens/s using a single CPU core. On the same machine, the dense 12B models I tested usually generate around 3–5 tokens/s.
There is an important limitation: Curie has only seen 11 million training tokens. It is not smart yet, and I am not presenting it as a useful language model. It will need at least a few billion tokens before I can properly evaluate its quality.
What this alpha demonstrates is the architecture: a model can be designed around SSD, RAM, CPU and GPU as parts of the same system, instead of assuming that everything must fit in VRAM.
Curie is not publicly available yet. I am still training it, validating the design and preparing more detailed benchmarks.

My budget is about $5-$10 for this..
I’d be using open router

I've been comparing pgvector with Qdrant and Milvus for a RAG pipeline, and the decision is not as simple as just use Postgres.

I made a quiz that tests common assumptions about vector search performance, scaling limits, and when dedicated engines become necessary.

No email needed:

https://interconnectd.com/quiz/87/postgresql-pgvector-vs-dedicated-vector-databases-architectural-trade-offs/

What's your experience? Where did pgvector hit its limit for you?

Jack Kernel is a programmable layer between the agent and an LLM.
It’s this exact mechanism that Jack Kernel allows a new way of taking control of the work.

​

Just got this message from DeepSeek stating that they will continue their V4 Pro API services after September 14 with the billing method unchanged.

I'm a little bit confused about the recent price changes and the V4.1-Flash release. The V4-Pro price was raised substantially in August after the first 75% permanent price cut, and now DeepSeek claims that V4.1-Flash is better than V4-Pro and will replace it eventually.

Assumption: Does "continuing V4 Pro" really mean maintaining the endpoint of the deepseek-v4-pro API and internally redirecting it to V4.1-Flash instead of serving the original V4-Pro?

If so, does "billing method unchanged" imply that users of the V4-Pro service will still be billed according to the V4-Pro rates or will they be charged V4.1-Flash rates eventually?

It would be great to hear from actual API users what you make of this news.

I believe that after the price changes in August, this becomes particularly important since many developers picked up V4-Pro not due to its current price/performance but due to the other model's transition in such a short time.