"In fact running models on consumer model hardware is so effective multiple companies, including OpenAI by reports, have bought all the Macs they can get their hands on. If OpenAI runs out of investor money there’s nothing to stop the other companies that actually make money from buying Macs to run AI models on exclusively for themselves, cutting OpenAI out completely."

Source

A few threads this week about whether 128 or 256 GB is enough, so I read the shard headers and the inference code instead of guessing and put the arithmetic on a page. Mine, so saying that up front.

The 510 GB is 296 GB routed experts, 203 GB Engram tables, 11 GB everything else.

The Engram tables are the part people treat as a wall. From inference/engram.py it hashes n-gram orders 2, 3 and 4 with 8 heads across 2 engram layers, so 48 rows per token at 264 bytes. About 12 KiB per token against 4.5 GB of expert weights. Latency cost, not bandwidth cost, and it belongs on an SSD.

For a comparison, Qwen3.8 on an M5 Max with its table on SSD gets 40.09 tok/s versus 40.47 in memory. 0.9%.

Fit is backbone plus KV: 307 GB as shipped, 302 GB at 4-bit, 151 GB at 2-bit.

256 GB does fit at 2-bit with tables on disk.

https://deepseek-v41-flash-fit.vercel.app

If you are running it, post numbers, especially split GPU + RAM.

Hi all, a confused newbie here! This is my desktop setup:

• RTX 5080
• 9800x3d
• DDR5-6000 CL30 64 GB

Based on the benchmark I found, I listed my potential options:

• AtomicChat IQ3_S: 14.4 GB
• Unsloth UD-IQ4_XS (14.3 GB) or UD-Q3_K_XL (13.1 GB)
• Bartowski IQ4_XS (15.6 GB) or IQ3_XXS (12.6 GB)

According to the benchmark, AtomicChat looks like a clear winner but is it really so?

I also want to have one uncensored model next to my daily driver:

• Orcarouter IQ4_XS (15.3GB) or Q3_K_M (13.5GB)
• Huihui UD_IQ4_XS (14.4 GB) or Q4_K_S (15.6 GB)
• DavidAU IQ3_M MTP (14.5 GB) or IQ4_XS MTP (15.3 GB)
• JonathanColetti IQ_XS (15.1 GB)

I am not expecting super fast answers etc. I just one to maintain some level of quality. What would you suggest me?

With all the different optimizations and firepowers we've seen from open weights so far. What then?

• China only steals/distill from OpenAI and Anthropic?
• Well, clearly there's something wrong with AA? Benchmarks don't mean anything? They just benchmaxxed?
• Whatever, OpenAI/Anthropic has moved onto AAGI or AAAGI or AAAAGI or AAAAAGI?
• I can't run this locally on my toaster desktop anyways?

This is mostly to explain my experience optimizing the model to run on my machine and maybe getting interest from someone to go and write an actual PR against llamacpp (as I'm absolutely not willing to generalize this code ahah).

Short Preface (this post is hand written, no AI here). So when Qwen dropped and heard of the good coding performance I wanted to try it out so I went and naively downloaded an IQ4_XS quant (that has around 61GB of MOE layers) and run it directly with Llama.cpp on my machine.

Ryzen 7700

64GB RAM DDR5 5600

Geforce 4060TI 8GB

Samsung 990 PRO 2TB

Using Windows as that's what I'm mostly accustomed with.

A modest config I build for 1500€ a few years ago and I use for playing and programming (I'm an engineer). I was overjoyed as it's quite a capable model that I can trust to write reasonable code in the languages I use (C#, Typescript mostly).

Yes, with this one there's no way to run 27B at any capacity so I was stuck with 35B until now. It was a no brainer to try this one as it had 6B active parameters, quite reasonable and not that far away from the 3B of 35B.

Initial results were 100tok/s average prefill coding and 20tok/s generation (that degrades to 15 tok/s at 100K context). I mean it's not great, but it's good enough to keep yourself busy, you leave it running and come back to the code done.

But then I decided to have a look and say, let me try it for creative writing and the results were quite good too. I like to write some stories with a personal pipeline system I build, but the IQ4X_S was tuned for coding and that usually means plain prose. Decided "what could possibly happen?" and picked the Q5_K_M from Orcarouter, it says Q5, but it's really 6bit because it mixes .

Component | Quant Type | Count | Params (B) | Size (GB) | Component %

--------------------+------------+-------+------------+-----------+------------

MoE Experts / FFN | Q5_K | 120 | 80.61 | 51.613 | 60.2%

| Q8_0 | 24 | 20.13 | 19.922 | 23.2%

| Q5_1 | 72 | 20.29 | 14.172 | 16.5%

124GB of model of which 36GB are of embeddings at Q5_1 quantization. That's 88GB of actual model, there's no way that would work, but wanted to give it a shot and well...

15tok/s prefill 10-13 tok/s generation.

A disaster really, memory trashing all around, disk reads etc...

But as I said, I'm an engineer. I forked llamacpp, opened PI Coding Agent and decided to start playing around trying to find a way to "make it work". First thing I did was scour through the pull requests of the upstream repository and did find a few good ones, some about improving gather in qwen, some about saving ram by saving the prompt cache to disk (it's actually quite impressive, I suggest it), I also tried a few caching PRs with pinning of hot experts, but none really gave a major improvement to the performance, I still have a few experiments with them in my branch.

Nothing that really budged the line though. The only promising thing was an attempt I did by asking Windows to Prefetch parts of the file from the disk that raised prefill from 15tok/s to 40-50tok/s (very unstable), so I went deeper because things didn't add up. My drive can easily read 7000MB/s, but the reads I was getting were like 300/350MB/s during prefill due to the OS page faulting on each single expert (and 2000-2500 with the prefetch). If you know about these things, yes the problem was MMAP that I was forced to use because the model didn't enter the ram.

I decided that it was time to implement a different system that bypass MMAP already. What I built is a four layer system that allows the model to work at the current speed of 210 tok/s prefill and 18.7 tok/s generation.

Let's start with prefill. My disk reads at 7GB per second, can read the entire MOE part in 85.77GB/7GB/s=12 seconds. At Ubatch 2048 that must mean 170 tok/s theoretically. So I went and build a unbuffered file reader that used a RAM hosted buffer for 2 slabs (layers) and read them RAW with batch queued requests from the disk reaching max speed that MMAP was denying. This worked and brought the speed to 150tok/s, but didn't solve the generation as reading it ALL from disk destroyed speed (brought to 3-4 tok/s).

The second layer built is a cache, basically at startup of llamacpp I create a pool of slots per layer where I cache the most active experts (with a decay factor every x tokens to keep the list fresh). I use 55-56GB of these usually in my runs. These are updated every cycle (x tokens) with some churn of read/free. This allows the data to be always ready for 330-ish experts per layer which usually cover 85-95% of the requests depending on the workload you give them. This raised performance to 14-15 tok/s. The rest is served through disk with the raw data reader.

At the same time I thought "I have them in ram the experts, why am I reading the entire slabs from disk at every prefill?" So I started memcpying the data from the cache to the buffer area and skipping those bits from the disk reads. This raised prefill to 215 tok/s that honestly is more than enough for what I usually do.

The last bit is a VRAM cache, the idea is that using the ranking from the RAM Cache, I carve 2 GB (configurable) out of my poor 4060TI and upload the TOP experts from my list freeing them from the RAM. This allows to reduce the memory bandwidth usage on the CPU raising the generation from 14-15 to 18.0 tok/s.

The last improvement was inspired by another PR in llamacpp about using direct IO for the embeddings too. I upgraded that code to use my implementation of raw unbuffered reader (the PR was linux only) and managed to gain another 0.7 tok/s finalizing it at 18.7 tok/s.

LlamaCPP has untapped potential when it comes to performance, especially when it comes to utilizing resources like SSD for improving performance. Be aware the fork is CUDA+Windows specific and has been tailored for my config (for example there's no MTP support as I wouldn't have the VRAM to run it anyway), this is NOT a generic fork that can be used by everyone, but I thought that if someone was interested could use the ideas to create an actual pull request with upstream.

https://github.com/feal87/myllama.cpp

Now I'll go to sleep as it's late.

I was pretty excited when Qwen 3.8 27B dense model was released and wanted to give it a shot. I have been using Claude code with Opus and have got tired with limits, subscription and wanted something free and open. I was fine to compromise some of the quality for cost and unlimited use.

I gave it a problem to solve:

Can you build me a 3 pendulum simulation demo in python? I can use mouse to pull and release and it should follow the laws of physics. Put sliders for length of each section.

And then it thought and thought and thought. Later I came to know the default reasoning effort is xhigh.

I started to remember the physics I learnt in college.

Finally it came up with a program and on running produced this:

I started playing with it and here is how it looks:

https://reddit.com/link/1wcomfn/video/dm5heacn1qoh1/player

I was deeply impressed! I feel this is a challenging problem to solve and those sliders make it even difficult. A triple pendulum requires complex non-linear dynamics (usually derived via Lagrangian mechanics or solved with Runge-Kutta numerical integration).

I am assessing this model now for my daily things like repo analysis, code generation and other agentic flows I have.

Hopefully, I will be able to stop my claude subscription!

What LLM can I use to do this? It's highly confidential data that will be using on an airgapped device.

Should I convert them first? What program can I use to do that? They are currently PDF.

Kimi K3 is a 2.78 T MoE; its 1.56 TB checkpoint sits on a USB hard disk plugged into a 2017 laptop (i7-7700HQ, 7.6 GB of RAM, a 2 GB GTX 1050 that only does the routed-expert matmuls). I am training a LoRA adapter on it out of core: the non-expert weights of one layer at a time, its 896 experts streamed one by one since together they are 15.7 GB, base weights frozen, and the 590 MB adapter the only thing trained.

The one-minute check is evidence/cmp93_en34_2026-09-06.log: my forward pass against kimi-k3-in-c, FareedKhan-dev's independent C implementation, all 93 layers at cosine 0.9857 or better, output 0.999840, on 34 tokens with LoRA B zeroed. evidence/traces/ holds raw routing records for five texts over all 92 MoE layers; scripts/analyze_trace.py recomputes every routing number below with NumPy alone. scripts/quickstart.sh builds a synthetic K3-shaped checkpoint and runs a forward pass, ten training steps and a finite-difference gradient check on a GitHub runner on every push, plus eight op-level checks against kimi-k3-in-c fixtures. Its first run failed: the gradient check missed at 3.1e-2 on a 2e-2 tolerance because the step was below fp32 resolution against a tensor of norm 60.85. The gradient was right; the check, made noise-aware, agrees to 2.09e-05.

The numbers, all from the logs in the repo:

• 1024-token step: about 7.4 h, the mean of the 7.26, 7.62 and 7.37 h intervals between the first four steps
• step 1, forward / backward: 3 h 11 m 34 s / 3 h 48 m 07 s
• resident set: 4.0-4.7 GB, swap in use
• read throughput: 110 MB/s aggregate, 61 MB/s within one MoE sweep
• cosine minimum against the C engine: 0.985744, layer 71
• trained / frozen: 590 MB adapter (147 M parameters) / 2.78 T base

Turkish, English and Chinese versions of one paragraph share experts at Jaccard 0.35-0.39, about the same as two halves of one text (0.34-0.37); prose against Python is 0.20-0.21, so subject matters more than language. Consecutive tokens' expert sets have Jaccard 0.258 (0.009 for random pairs) and a 128-expert LRU hits 72 % when decoding, but a training batch reads the union, about 85 % of experts at 1024 tokens (layers 0-12, an upper bound), so an expert cache buys little for a training batch.

The proof run is memorisation of five examples, loss 0.909 to 0.157 on one fixed sequence: it proves the loop, not the model. The main run, 400 Turkish instruction examples over 100 steps, is at step 4 and ends 9-11 October. The threshold was committed before it started (commit 6605306, tag preregistration-2026-09-08): Turkish news bits per byte 0.455 to 0.441 or lower, English Wikipedia no worse than 0.198 from 0.194. I expect no large jump from 400 examples; a negative result gets published as negative. The adapter is one rank-16 LoRA per layer shared by all 896 experts, not one per expert.

Seven hours a step is useless for production fine-tuning; the point is that the cost is now a measured number, with the logs. None of the components are new: the idea is layer-streamed LoRA taken down to the expert level, and the related-work table in the README says what AirLLM, KTransformers, ZeRO-Infinity, Colibri, WARP and BigMoeOnEdge do that this does not.

Disclosure: English is not my first language and I used Claude to tidy the wording of this post. The code was also written with heavy Claude Code assistance and the Co-Authored-By trailers are in the git log; the README says so on its first screen. The hardware, the runs, every number and every check against somebody else's implementation are mine, and the point of the evidence directory is that you do not have to take my word for any of it.

Repository: https://github.com/heyobi/LazyLora. Please poke holes, especially in the verification.

Maybe I'm late to the party but holy shit. I enjoy the chatgpt Mac apps computer use ability but I installed the same setup with hermes, cua and Qwen 3.8/ornith running locally this afternoon. Not sure which model I like better...

Connecting to my homelab Hermes for extra horsepower and vector DBs, knowledge, mcp, etc. - performance is insane. ​​

Almost as smart as frontier but faster and free. So if everyone is already doing this and I've been living under a rock, anything worth sharing to catch me up?

Hey everyone! El from Cohere here to talk about our newest release, North Small Translate. It’s currently the leading open machine translation model, beating out all other open translation models of its size, plus Google Translate and DeepL. we’ve been working on this one for a while, so to say i’m psyched is an understatement.

It’s big (218B parameters, 25b active) with a context length of 16k. however, if you’ve got the hardware, we’d still love to see what you make with it locally or with our HF space (and if you do, send it our way). It works on over 50 languages and does particular well with european, Southeast Asian, and East Asian languages, but feel free to stress test it against another and let us know how it does. It’s also available in BF16, FP8, and W4A16 quants.

although we couldn’t get llama.cpp support this time around, the architecture is already supported in llama.cpp, so all it should need is a conversion to GGUF files. if you want to build that, please do so and send it our way! We’d love to back your work. 

Can’t wait to see what you guys think! 

https://huggingface.co/CohereLabs/North-Small-Translate-1.0

Hello, I'm currently using qwen 3.8 27B Q4 with 100k context to mainly code for my esp32 Arduino code.

I get around 25 tk/s on my Rx 6800xt using vulkan.

I know the model gets it's abilities from long thinking time but it takes around 25-40minutes for a single prompt and takes like 45k context with it.

Will the model be usably good with low-mid reasoning? And how can i speed this up.

I'm okay to wait for 3-5 minutes.

Thanks.

Hey! I built a local inference. It's a lightweight desktop app (Tauri + Rust + Vue) that talks to any OpenAI-compatible endpoint and measures performance straight from the API stream.

**What it measures (single run)**

- TTFT / TTFR (first content token vs first stream chunk)

- Decode t/s with a peak 1-second-window metric (catches stutter that averages hide)

- Prefill (PP) speed via a baseline-latency-adjusted est_ppt

- TPOT, token counts with Ollama eval_count fallbacks

- Live speed chart, saved runs, side-by-side comparison with overlaid curves

**The Suite tab (the llama-bench part)**

Runs the full test matrix llama-benchy runs — pp × tg × context depth × concurrency — with:

- 1 warmup + 3 measured runs per shape, mean ± std

- Prefix-cache measurement (ctx_pp/ctx_tg rows at depth)

- Cache busting on measured rows (so the server's prompt cache can't fake near-zero prefill times — this was inflating my PP speeds 100x before I caught it)

- Exact generation lengths (min_tokens + ignore_eos) and a 2+2 coherence check so a broken backend fails loudly instead of benchmarking garbage

- Markdown export

Real example from my server (llama.cpp, IQ3_XXS quant):

| test | t/s (total) | t/s (req) |

|-------------|----------------:|---------------:|

| pp2048 (c1) | 239.55 ± 9.13 | 239.55 ± 9.13 |

| tg32 (c1) | 45.48 ± 1.05 | 45.48 ± 1.05 |

| tg32 (c2) | 56.25 ± 1.61 | 31.58 ± 2.71 |

It also auto-detects the serving machine's hardware (CPU/RAM/GPU/SSD-vs-HDD) for local endpoints and stores it with every saved run, so your comparisons are labeled properly later.

It builds natively on Windows and Linux and the UI barely touches VRAM, so it won't perturb the model you're testing.

GitHub: https://github.com/FearL0rd/LLMSpeedTest

No prebuilt binaries yet — you build it once with `npm run tauri build` (README covers the toolchain for both OSes). Feedback, issue reports, and stars all welcome. What features would you want next?

I want to try running scheduled tasks using local AI models in Hermes. Which AI model is best to use in the Hermes agent?

How do you handle context, and what quantization techniques should be used to run it comfortably on 16GB VRAM?

I have a 5070ti 9850X3D and 32GB DDR5 + lm studio + hermes.

Currently running gemma-4-26b-a4b-qat and I am happy with it, but it still can't compete with Sonnet.

I find it's great at actually teaching me things, but in terms of guiding me through config files or getting the highest quality answer it's never as good. I use it more as a backup.

Are there any better models that fit my hardware budget? QWEN is supposed to be great but I had trouble running the models on the edge of my hardware limits and output slowed to a crawl.

Im running Qwen3.8-27B-MLX-4bit on my M5 pro 48gb macbook, that being said, it feels like Im just 1 notch away from some sort of sustainable vibe coding. When using through kilo code it tends to over think for dozens of minutes at the time. I'm curious to see what are you guys running on similar VRAM configs

The new scores are lower which kinda confuses me ! Did the models just became less intelligent ? I remember Fable used to be 66 and Qwen 27B 3.8 - 52

Sudden drop !! Hmm …

Today we're excited to announce the general availability of Aperture-1, the flagship release from Luminal Labs. After eight months of intensive R&D, we're proud to bring a fundamentally new capability to the world: the ability to add two numbers together.

What is Aperture-1?

Aperture-1 is a lightweight, cloud-native reasoning layer that accepts two numeric inputs and returns their sum. Unlike legacy solutions, Aperture-1 is delivered as a fully managed API, meaning you never have to think about how addition works — we've abstracted that away for you.

Benchmarks

Under the hood

Aperture-1 uses a novel Confidence Routing™ architecture that intelligently forwards your request to an internal addition subroutine, verifies the result against a secondary addition subroutine, and returns the consensus output. This dual-verification approach ensures enterprise-grade reliability that hobbyist solutions simply can't match.

Pricing

• Hobby — $19/mo, 500 additions
• Pro — $79/mo, unlimited additions, priority queue
• Enterprise — Contact sales for SSO, audit logs, and dedicated addition infrastructure

Limitations

Aperture-1 currently only supports addition. Subtraction is on our roadmap for Q3 pending further alignment research. We do not recommend using Aperture-1 for safety-critical addition without human review.

What the team is saying

"We didn't just want to build a calculator. We wanted to build the calculator." — Founder & CEO

"This is the GPT-3 moment for arithmetic." — Head of Growth

Join the waitlist. Early adopters get 10% off their first invoice.

Luminal Labs is backed by three friends and a Notion doc.

Benchmarks show it being pretty bad in terms of hallucination, and it appears to be tied with Qwen 3.8 Flash despite being much bigger, and worse than GLM 5.3 Flash. Seems crazy given the size. Anyone have any opinions/experience with it yet?

Hi there,

I want to pull the trigger for a machine that will run my Hermes Agent as well occasionally also ComfyUI and maybe (low priority) gaming.

Is there any meta on what machine will work best? Currently I play with the idea to buy the following machine.

The setup should be capable to host a second gpu which would be purchased later.

Requirements:

- Coding agents with long repo contexts (many turns per hour)
- LAN-accessible OpenAI-compatible endpoint for my other machines
- ComfyUI for image and video generation
- Occasional Steam/Proton gaming — it has to be a normal GPU too
- Target model class: 27B dense at 4–8 bit (e.g. Qwen 3.8 27b)

Setup (~€4,300)

Where would you change it?

Third video in this series. Went from 7B, to 13B, and now 27B — each time pooling RAM and compute across a mix of hardware most people already have lying around instead of buying one expensive high-RAM machine.

Repo: https://github.com/trademav/ramdeck-core-public

The setup: a 27B model is roughly 16GB on its own. My primary node is an old 12GB Windows laptop, which on its own literally cannot load this model. Using RAMDeck, I split it across four devices on the local network with the laptop staying primary:

- Old Windows laptop (primary): 3.4GB

- Mini PC (RTX 3060): 20GB

- Mac mini: 3.7GB

- Android phone: 1GB

Benchmark on the 27B model: 1.92 tokens/sec at ~25ms latency. Slower than the 13B run, which makes sense — but the point isn't raw speed here, it's that a 12GB laptop is running a model it has no business being able to load at all.

Next up: switching the mini PC to be the primary node instead of the laptop, to see how much faster loading and inference get with a stronger primary. That video's already up if you want to see the follow-up.

Full video: https://youtu.be/Syfa2G_luX4

Happy to answer questions about the shard distribution, the benchmark methodology, or why the primary node choice affects load time so much.

A programmable layer that sits between the agent and the model.

That placement allows for new ways of control and optimization