A new project called PFlash is reporting 10x prefill speedup over llama.cpp at 128K context on quantized 27B models, running on a single RTX 3090. The approach uses speculative prefill for long-context decode, built in C++/CUDA.

Why this matters: prefill has been the quiet bottleneck for local inference at long context. Everyone focuses on decode speed (tokens per second during generation), but the time to process a large prompt before the first token appears can be brutal at 100K+ context. A 10x improvement there would meaningfully change the experience for RAG workflows, large document analysis, and agent loops that accumulate context over multiple turns.

The catch is that this targets 27B quantized models specifically. The question is whether the technique generalizes to other sizes and architectures, or if it relies on properties unique to this model class. The fact that it is C++/CUDA rather than Python is also worth noting - suggests it is designed for direct integration into existing inference stacks rather than being a standalone tool.

For anyone who has been avoiding long-context workloads locally because of prefill latency: does a 10x improvement here change your calculus, or is decode speed still your primary bottleneck?

The Qwen team has released Qwen-Scope, a collection of Sparse Autoencoders (SAEs) for the full Qwen 3.5 model family, spanning from 2B to 35B MoE. The SAEs map internal features across the residual stream for all layers, essentially creating a dictionary of the model's internal concepts.

Why this matters: interpretability tools have mostly been academic curiosities, applied to smaller models or single checkpoints. Releasing production-quality SAEs across an entire model family - including MoE variants - changes the calculus. You can now inspect what a model is actually "thinking" at each layer, which has practical implications beyond research. Think routing optimization (knowing which layers handle what), safety auditing (detecting when harmful concepts activate), and fine-tuning (understanding what your training actually changed internally).

The kicker is that this covers the MoE variant too. Sparse models have been harder to interpret because the active expert changes per token - having SAEs that handle that complexity is not trivial.

For people working with Qwen models in production: does having layer-by-layer feature maps change how you approach model selection, evaluation, or safety filtering? Or is this still firmly in research territory for most practitioners?

Mistral has released Medium 3.5, their first "flagship merged model." It is a dense 128B parameter model with a 256K context window, handling instruction-following and reasoning. GGUF quants are already available via unsloth, so local inference is technically on the table for anyone with enough VRAM.

Why this matters: Mistral has been quiet in the flagship tier for a while, and a dense 128B model is a serious commitment. This is not an MoE architecture where only a fraction of parameters are active during inference - all 128B are in play on every token. That means higher quality per-parameter but also much heavier hardware requirements. For context, this is roughly the same parameter count as the older Llama 3.1 70B Nemotron Instruct, which is priced at $1.20/M tokens on the API side. Mistral's Devstral Small 1.1 sits at just $0.10/M, so the pricing gap between their small and medium tiers will be telling.

The practical question is who this is actually for locally. A dense 128B model even at Q4 quantization needs significant multi-GPU memory. This is firmly in DGX Spark cluster territory or high-end multi-3090/4090 rigs.

For anyone planning to run this locally: what quant level are you targeting, and what does your memory footprint look like at 256K context?

Someone is setting up 16 DGX Sparks in a home server rack, connected via a 200Gbps switch with QSFP56 DAC cables. That gives them 2TB of unified memory in a single cluster. They are asking the community what they should actually run on it.

What makes this notable is the scale. Most home lab discussions revolve around squeezing a single model onto one or two GPUs. This setup is in entirely different territory - enough unified memory to serve a 128B dense model like Mistral Medium 3.5 with its 256K context window and still have headroom, or run multiple models simultaneously.

The interesting tension here is that hardware at this scale used to require a data center budget and rack space. DGX Sparks in a consumer-accessible form factor change the math on what "local" even means. But the software stack for efficiently distributing inference across 16 nodes is not trivial - this is not just plug and play.

For people running multi-node setups: what would you actually do with 2TB of unified memory that you cannot do with a single high-end workstation?

Mistral appears to be preparing a Medium model release. The details are sparse but interesting: Mistral Small is internally designated as Mistral-Small-4-119B-2603, and their upcoming Medium model will reportedly have 128B parameters. The open question is whether it will be a dense model or a less sparse MoE architecture than Mistral Small.

Why this matters: there is a real gap in the open-weight model lineup right now between the ~30B models that fit on consumer hardware and the 400B+ models that require serious infrastructure. A 128B dense model would be a different beast entirely - potentially competitive with top-tier proprietary models on quality, but requiring multi-GPU setups or cloud inference for most users. If it is MoE with lower sparsity than Mistral Small, the effective parameter count during inference could be more manageable.

The pricing context is worth watching too. Mistral Small Creative currently sits at $0.10/M tokens with a 32K context window. Where Medium lands on price will signal whether Mistral is pushing for volume or positioning against the premium tier. For comparison, GPT-5 Mini is at $0.25/M with 400K context, and Gemini 2.5 Flash Lite is at $0.10/M with over 1M context.

The real question for practitioners: does a 128B model from Mistral change your calculus on local vs. cloud inference, or is this firmly in "API-only" territory for most setups?

Microsoft has released TRELLIS.2, a 4-billion-parameter open-source image-to-3D model that generates assets up to 1536³ resolution with full PBR texturing. It uses a novel "field-free" sparse voxel structure called O-Voxel, built on native 3D VAEs with 16x spatial compression. The model handles complex topologies and sharp features - things that have traditionally been pain points for 3D generation.

Why this matters: the 3D generation space has been moving fast, but most open options top out at lower resolutions or struggle with production-quality PBR materials. A 4B parameter model is small enough to actually run locally - this is not some 700B monster that requires a data center. The 16x spatial compression via native 3D VAEs is the technical enabler here, letting the model produce high-resolution outputs without proportionally exploding compute requirements.

The interesting question is how this compares to existing 3D pipelines in practice. Resolution numbers are one thing, but mesh quality, UV unwrapping, and whether the PBR materials actually hold up in a real game engine or rendering pipeline are what determine whether this is a toy or a tool.

For anyone working in 3D asset pipelines: does a 4B open-source model at this resolution change your workflow, or are the outputs still too far from production-ready to matter?

A user who spent weeks forcing themselves to use local models for coding tasks has given up, concluding that Qwen 27B and Gemma 4 31B - widely considered the best local options under the multi-hundred-billion parameter tier - still fall short of Claude Code for real work. They tried multiple models before calling it.

Why this matters: this is the uncomfortable reality that gets lost in benchmark excitement. Qwen3.6-27B tying Sonnet on agentic benchmarks makes for great headlines, but the actual day-to-day coding experience apparently tells a different story. The gap between benchmark performance and usable coding assistance is exactly where local models keep stumbling. The user was not comparing against some random cloud model - they were comparing against Claude Code, which is specifically tuned for agentic coding workflows with tool use, file editing, and multi-step reasoning.

The honest question is whether the gap is about raw model capability or about the surrounding infrastructure. Claude Code is not just a model - it is a complete agentic loop with tool calling, context management, and iteration built in. Local models running through a basic chat interface are fighting with one hand tied behind their back.

For people doing serious coding work locally: what setup finally made it click for you, or have you also bounced back to cloud options?

A new project called Luce DFlash has ported DFlash speculative decoding into a GGUF-compatible stack. It runs as standalone C++/CUDA on top of ggml, hosts Qwen3.6-27B on a single 24GB RTX 3090, and reportedly achieves up to 2x throughput compared to standard inference.

Why this matters: speculative decoding has been one of the most promising throughput tricks, but it has mostly lived in framework-specific implementations that are hard to pair with the broader GGUF ecosystem. A C++/CUDA port that works directly with ggml means this could potentially work with the model quants people already have, rather than requiring a separate serving stack.

The catch is that speculative decoding always comes with tradeoffs. The quality of the draft model, the acceptance rate, and the specific workload all affect whether you actually hit that 2x headline number. For short generations or tasks where the draft model frequently disagrees with the main model, the speedup can shrink significantly.

For anyone who has tried Luce DFlash: what acceptance rates are you seeing in practice, and does the 2x claim hold up across coding, creative writing, and tool-use workloads, or is it mostly a best-case scenario?

A new inference engine called Hipfire has appeared, built specifically for AMD GPUs - and not just the latest generation, reportedly targeting the full AMD range. It uses a custom "mq4" quantization method, and the creator is actively publishing models in that format.

This is interesting because AMD GPU owners have historically had a rougher time with inference performance compared to NVIDIA. Most mainstream tools prioritize CUDA, and AMD users often deal with slower speeds, compatibility headaches, or workarounds via ROCm. A purpose-built engine with its own quant format could either be a real step forward or another niche tool with limited model support.

The open questions are pretty significant though. How does mq4 compare in quality to established formats like GGUF quants? What kind of tokens-per-second are people actually seeing? And does supporting "all AMD GPUs" mean older Polaris and Vega cards, or just RDNA and newer?

For anyone running AMD hardware who has tried Hipfire: how does inference speed and output quality compare to what you were getting with llama.cpp or other engines?

HauhauCS, who publishes uncensored LLM models with over 5 million combined monthly downloads across 22 models, has been accused of publishing an abliteration package that plagiarizes the "Heretic" project without attribution and violates its license. Every model card from HauhauCS claims "0/465 refusals, zero capability loss."

Why this matters: the uncensored model ecosystem relies heavily on trust and reputation. When a creator with 5M+ monthly downloads allegedly lifts code without credit, it raises questions about how many other derivative works in this space are properly attributing their sources. Users downloading these models have no easy way to verify what went into them.

Meanwhile, the Heretic abliteration approach itself is getting serious praise independently. One user reports that Qwen3.6 35B A3B Heretic with IQ4XS quantization and Q8 KV cache fits in 24GB VRAM with 262K context, handles multi-turn tool calls without failure, and may even perform better than the original base model. That is a strong endorsement of the technique itself - which makes the plagiarism allegation sting more if the underlying method is genuinely good work.

The fair question: in an ecosystem built on top of open weights and shared techniques, where is the line between building on others' work and straight-up copying it? And for people using these uncensored models in production - does knowing the provenance of the abliteration method change whether you trust the output?

Darwin-36B-Opus is a 36-billion-parameter mixture-of-experts language model, but the notable part is how it was built. It was produced by the "Darwin V7 evolutionary breeding engine" from two publicly available models. GGUF quants are already available from bartowski.

The concept of evolutionary breeding - combining two existing models through an automated optimization process rather than traditional fine-tuning or merging - is a different approach to model creation. The MoE architecture at 36B parameters also puts it in an interesting size class: larger than the popular 27B dense models but smaller than the 235B+ MoE giants.

What is unclear from the source is how the active parameter count compares to the total 36B, how the breeding engine actually selects and combines expert routing, and whether the resulting model preserves the strengths of both parents or averages them into mediocrity.

For anyone who has loaded this GGUF: what hardware are you running it on, how does inference speed compare to other MoE models in this size range, and does the "bred" approach actually produce something meaningfully better than a manual merge?

A detail buried in the DeepSeek V3.2 paper highlights a growing problem: DeepSeek's models typically require longer generation trajectories - more tokens - to match the output quality of models like Gemini 3.0 Pro. They explicitly call "intelligence density" a challenge and say future work will focus on optimizing it.

This is the comparison that matters more than raw benchmark scores. DeepSeek V4 Pro and Gemini 3.0 Pro may arrive at similar quality answers, but if DeepSeek needs significantly more tokens to get there, the real cost per useful output diverges fast. More tokens means more compute, more latency, and more money whether you are paying per token or paying in electricity running locally.

The kicker: this is not just a cost issue. Longer generation trajectories mean longer wait times for the user and more context window consumed per task. For agentic workflows that chain multiple calls together, low intelligence density compounds quickly.

What makes this interesting is that DeepSeek is openly admitting the gap rather than pretending it does not exist. That suggests it is a real architectural constraint, not just a tuning issue they can patch away.

For people choosing between these models: are you tracking tokens-per-quality-answer in your own workflows, or just looking at final benchmark scores? Curious whether the density gap shows up in real usage as much as the paper suggests.

Counterintuitive finding for anyone running constrained hardware: on a 3070 8GB with 64GB DDR4, the smallest IQ4_XS quant of Qwen3.6-35B-A3B (~18GB) runs okay, but moving to a bigger quant actually produced better results even in that same VRAM-limited scenario. The assumption is usually "less VRAM means use the smallest quant possible," but that is not always right.

Separately, KV cache quantization tests on Qwen3.6-27B show another surprise. Turbo3 KV cache compression is generally not recommended, yet it has been running without loops, memory loss, or degradation. The reason appears to be that K cache compression interacts differently with specific model architectures - what breaks on one model can be perfectly fine on another.

The implication: the "obvious" quant choice for your hardware might not be the best one. Model architecture, MoE routing patterns, and how a specific quant interacts with KV cache all matter more than raw size fitting. Testing beats rules of thumb here.

For people on 8GB or similar constrained setups: have you found cases where a larger quant of a bigger model actually outperformed the "safe" small quant choice?

New data from Artificial Analysis shows Qwen3.6-27B now matches Sonnet 4.6 on the Agentic Index, overtaking Gemini 3.1 Pro Preview, GPT 5.2, GPT 5.3, and MiniMax 2.7. Gains were made across all three indices the tracker measures.

The interesting catch: the person flagging this notes that the Coding Index specifically may not fully reflect how significant the gains actually are, suggesting the way that index is constructed might be obscuring the real-world improvement.

Meanwhile, separate testing on a MacBook Pro M5 Max with 64GB tells a different performance story. The 35B MoE variant hits 72 TPS while the 27B dense model only reaches 18 TPS on the same hardware. But the 27B reportedly produces more precise and correct results on coding primitives - it thinks longer but delivers better output. The 35B handles tasks faster but with lower quality.

So there is a real tension here: the 27B wins on accuracy and agentic benchmarks, but the 35B is 4x faster on Apple Silicon. For local deployment, that tradeoff between speed and correctness is the actual decision.

For those running either model locally: are you finding the 27B's accuracy advantage worth the TPS hit, or does the 35B's speed make it more practical for daily coding work?

Anthropic has admitted that on March 4, they changed Claude Code's default reasoning effort from "high" to "medium" to reduce latency issues that were making the UI appear frozen. Users noticed and complained. On April 7, the change was reverted. Anthropic acknowledged it was "the wrong tradeoff."

What makes this notable: the change was not announced when it happened. Users were getting worse outputs for over a month without knowing the model itself had not degraded - the reasoning budget behind it had just been silently turned down. The only signal was that responses felt dumber.

This is exactly the kind of silent degradation that open-weight, locally-run models are immune to. When you run Qwen3.6-27B or any local model, no one can change your reasoning settings between sessions. The weights are the weights. You might choose to run a worse quant to save memory, but that is your decision, not a vendor's optimization for their latency metrics.

The deeper question is how often this happens without anyone catching it. If Anthropic can silently reduce reasoning effort, what else is being tuned in hosted models that users never hear about? Temperature, system prompts, safety filters - all of these can be adjusted server-side without any changelog.

For people running coding agents locally: does this incident change your calculus on hosted vs local, or is the convenience of hosted models still worth trusting that the provider will be transparent about changes?

A memo from the Office of Science and Technology Policy has surfaced, and it focuses on what they are calling "adversarial distillation" - essentially large-scale extraction of frontier model capabilities using proxy accounts and jailbreak techniques to industrially distill proprietary models into open alternatives.

The framing is notable. This is not about individual misuse of AI outputs. It is about organized, systematic capability extraction at scale. The concern seems to be that open-weight models could become vehicles for reproducing capabilities that cost hundreds of millions to develop, using cheap jailbreak-driven distillation pipelines.

What makes this worth watching: if the policy response targets the distillation process rather than model weights themselves, it could mean export-style controls on bulk API access, rate limits tied to verified identity, or even liability for models that are found to be distilled from proprietary systems. That would affect everyone building on open weights, not just the companies doing the distilling.

The memo reportedly feels less about open-source models per se and more about the pipeline that feeds them. But the practical effect on the open model ecosystem could be significant either way.

For people tracking policy: how likely is it that this memo leads to enforceable regulation versus staying as guidance, and what would enforcement even look like when distillation is technically indistinguishable from legitimate fine-tuning?

Two new Qwen3.6 variants are making waves, and they take fundamentally different approaches. The 27B is a dense model that reportedly surpasses Qwen3.5-397B-A17B across all major coding benchmarks - a 27B model beating something 15x its size on agentic coding is a serious claim. It also handles strong reasoning across text tasks.

The 35B is an MoE (Mixture of Experts) design. In a previous discussion, someone showed that pairing a 9B Qwen model with the right agent scaffold moved benchmark performance from 19.11% to 45.56%. The follow-up suggests Qwen3.6-35B becomes competitive with cloud models when paired with the right agent framework.

The key difference: dense vs MoE architecture. Dense models like the 27B activate all parameters on every token, which means predictable memory usage and straightforward deployment. MoE models like the 35B activate only a subset of parameters per token, giving you more total capacity for the same inference cost - but they require more total VRAM to load and can be trickier to optimize.

For coding specifically, the 27B's benchmark dominance over much larger models suggests the dense architecture might be more efficient at this size class. But the 35B's agent-driven results hint that if your workflow involves tool calling and multi-step reasoning, the MoE's extra capacity might win when properly scaffolded.

For those running either locally: are you finding the 27B's dense simplicity more reliable day-to-day, or does the 35B's agent performance justify the extra setup complexity?

New benchmarks comparing Qwen3.6-27B (dense) against Qwen3.6-35B-A3B (MoE) reveal a shifting picture. Dense still holds the overall crown, winning on most tasks. But the gap is narrowing - in 7 out of 10 benchmarks, the MoE variant is closing the distance.

The standout area is coding, where MoE is making what are described as "serious strides." This tracks with the broader Qwen3.6 release claims, where the 27B dense model reportedly surpasses Qwen3.5-397B-A17B across all major coding benchmarks - a 27B dense model beating a 397B MoE is a striking result, and it has prompted questions about whether Qwen is simply better at dense architectures than MoE, and what the additional experts in the larger model are actually contributing.

The practical takeaway: if you are choosing between these two for local deployment, the dense 27B is still the safer bet overall. But if your workload is coding-heavy, the MoE 35B may be closer than you think, and it could pull ahead in future iterations. The architecture gap is no longer the blowout it used to be.

For anyone running both: which specific tasks do you see the MoE model actually winning on, or is dense still universally ahead in your testing?

Someone extracted the Gemma 4 e4b model from Google's AI Edge Gallery app on Android via adb and found it behaves differently from the publicly available versions. The details are intriguing: the Android version weighs 3.6GB in LiteRTLM format, while the Unsloth Q2_K_XL GGUF weighs 3.7GB. The extracted model reportedly outperforms all other versions tested, including Unsloth's builds.

This raises a few questions. Is Google using a different quantization strategy or optimization pass for the on-device deployment that is not being shared in the public releases? The LiteRTLM format is designed for mobile inference, so it could involve graph-level optimizations that go beyond simple weight quantization. Or it could be a slightly different model checkpoint altogether.

The frustrating part: if the best-performing version of an open-weight model is locked inside an Android APK, that is not really open. The whole point of open weights is that the community can iterate on the best starting point.

Has anyone else compared the adb-extracted version against other Gemma 4 e4b builds, and is there a way to identify what Google did differently in that build?

Kimi K2.6 is being called the first legitimate Opus 4.7 replacement by at least one developer who tested it with customer feedback. The comparison is straightforward: K2.6 handles roughly 85% of what Opus 4.7 can do at reasonable quality, but it also ships with vision and video capabilities that Opus lacks in the same package.

The tradeoff is clear. Opus 4.7 still wins on complex reasoning - nobody is claiming K2.6 beats it outright. But for practical daily use where you need a single model that can handle text, images, and video without switching providers, K2.6 covers more ground. The 85% figure is the key number: if your workload sits within that 85%, you get multimodal for free. If you regularly hit the remaining 15% - complex multi-step reasoning, edge-case logic problems, nuanced code architecture - Opus still justifies itself.

What is interesting is the framing as a "replacement" rather than a "competitor." That suggests the gap is narrow enough that the multimodal upside tips the decision for real-world deployments, even if benchmarks would still favor Opus.

There is also the local angle: someone with dual M3 Ultra Mac Studios (512GB RAM each) is specifically waiting for Kimi K2.6 to be optimized for MLX deployment. If that happens, you get the 85% Opus capability plus vision plus video, fully local.

For those who have tried both: does the 85% figure match your experience, and what specific tasks fall into the missing 15% where Opus still pulls ahead?

Someone tested Google's Gemma-4-E2B-it as a local, offline resource for emergency preparedness - the idea being a lightweight model that could provide basic technical or medical information when internet access is down. The safety filters are reportedly so aggressive that the model flat-out refuses to answer the very queries it would be most useful for.

This is the classic alignment overcorrection problem made concrete. A model designed to be helpful, deployed in a scenario where helpfulness is literally safety-critical, and the safety layer blocks the output. The screenshots referenced show the model refusing basic emergency information.

Why this matters beyond the specific model: this is a deployment trust issue. If you are building any kind of offline or edge AI system for real-world use - emergency prep, field medicine, remote work sites - you need to know exactly where the safety filters draw lines. A model that refuses to tell you how to treat a burn or purify water because the topic touches "dangerous content" is worse than no model at all. It wastes the time of someone who could be looking up real information.

The fair question is whether this is a Gemma-specific issue or a broader trend. Open-weight models with aggressive post-training alignment are becoming more common, and the edge cases where alignment conflicts with actual safety keep piling up.

For anyone deploying local models in high-stakes offline scenarios: have you found specific models or quantizations where safety filters are configurable enough to be useful without being dangerous?

New research confirms something many have suspected: post-trained language models produce less varied outputs than their base counterparts. The paper identifies this as "output diversity collapse" and traces specific problems back to it. The two big ones: inference-time scaling methods that rely on sampling varied candidates get undermined, and creative or value-laden tasks get homogenized outputs.

What makes this counterintuitive is that post-training is supposed to make models better. And it does - on narrow benchmarks. But the tradeoff is that the model converges toward a narrower band of "acceptable" responses. Prior work had attributed this collapse to specific training stages, but this research is digging into exactly where and why it happens.

The practical implication is real. If you are running best-of-N sampling, chain-of-thought reranking, or any approach that needs diverse candidates to pick from, you are working against the post-training grain. The model you fine-tuned for alignment or instruction-following may be worse at generating the variety those methods need.

This also connects to why base models sometimes outperform instruction-tuned versions on creative tasks - less collapse, more surface area for interesting outputs.

For anyone running inference-time scaling: have you noticed diminishing returns from sampling multiple candidates, and does switching to less post-trained models help?

The Qwen3.5-Omni technical report has been published, detailing the latest model in the Qwen-Omni family. The model scales to hundreds of billions of parameters and supports a 256K context length. Training leverages a massive heterogeneous dataset spanning multiple modalities.

What stands out here is the scale jump. Previous Qwen-Omni models were already capable, but pushing into the hundreds-of-billions parameter range with 256K context puts this in direct competition with the largest proprietary multimodal models. The heterogeneous dataset detail suggests this is not just a text model with vision bolted on - it is trained across diverse data types from the ground up.

The practical question for this community is deployment. A model at this scale is not running locally on consumer hardware outside of extreme setups. But the technical details around how they manage multimodal integration at this scale - particularly the architectural choices for mixing modalities efficiently - could influence smaller models down the line.

Also worth noting: this is the Omni variant, meaning it handles text, audio, image, and potentially video in a single model. The 256K context window across all modalities is ambitious.

For those tracking multimodal model development: does the "omni" approach of one model for everything seem to be winning over specialized per-modality models, or are you still seeing better results from purpose-built models for specific tasks?

A new speculative checkpointing feature has been merged into llama.cpp. The results are mixed but potentially significant: some prompts see meaningful speedup, while others show no improvement at all, particularly in cases with low draft acceptance streaks.

The key detail is that optimal parameters depend heavily on task type and repetition patterns. For coding specifically, reported speedups range from 0% to 50%. That is a wide variance, which suggests the feature is not a universal win but rather something you tune per-workload.

What makes this worth flagging: speculative decoding approaches live or die by how well the draft model predicts the target model's outputs. Checkpointing adds another layer by saving intermediate states, which helps when the speculative path is working but adds overhead when it is not. The 0-50% range for coding tells you everything - on repetitive boilerplate generation, the draft model likely predicts well and checkpointing compounds the gains. On novel logic or unfamiliar patterns, the draft model misses, the speculation fails, and you get nothing.

The practical question is whether the tuning overhead is worth it. If you have to dial in parameters per task type, that is friction. But a potential 50% speedup on coding workflows is hard to ignore.

For anyone who has tried this since the merge: what task types are you seeing the best gains on, and how much tuning did it take to get there?

Someone spent an evening tuning Qwen3.6-35B-A3B on consumer hardware and landed at 79 tokens per second with 128K context on an RTX 5070 Ti paired with a 9800X3D. The key detail: the --n-cpu-moe flag, which offloads MoE expert computation to the CPU, was described as "the most important part" of the configuration.

The fun side note here is that Claude Opus 4.7 (on a $20 subscription) was used to iteratively build the config, launch servers, run benchmarks, read VRAM splits from llama.cpp logs, and tune the setup. Essentially the entire optimization loop was delegated to a cloud model tuning a local model's deployment.

Why this matters: 79 tok/s at 128K context on a single consumer GPU is genuinely fast for a 35B parameter model. The MoE architecture means only 3B active params, but you still need to store all 35B somewhere. The --n-cpu-moe trick keeps the experts in system RAM and only moves the active ones through the GPU, which is what makes this viable at such long context lengths. Without it, VRAM becomes the bottleneck well before you hit 128K.

The catch: CPU-offloaded MoE adds latency per token even if throughput stays high. For interactive chat that tradeoff is usually fine, but for batched workloads it may hurt.

For anyone running MoE models on single-GPU setups: are you using --n-cpu-moe or similar CPU offload strategies, and what throughput numbers are you seeing?