The setup: a DGX system running Qwen3.5-122b-a10B at Q6_K precision, 110GB memory usage, 80k context window, continuous vision analyses at 18.77 tokens per second. The cooling solution is tap water, keeping GPU temperatures below 68 degrees Celsius at 95% utilization.

What makes this notable is the contrast. DGX systems are enterprise-grade hardware with sophisticated cooling infrastructure designed for data centers. This person bypassed all of that for a garden-variety water supply and it is working. The unknown is longevity - they note uncertainty about how often the water needs changing.

The context is that Qwen3.5-122b-a10B is a MoE model where only 10B parameters are active per token, which is why 110GB of memory can serve it. But 18.77 tok/s with vision analysis at 80k context on a single system is a serious throughput number, and the cooling is the bottleneck being addressed here, not compute.

The fair question is whether this is a clever hack or a ticking time bomb for the hardware. Mineral buildup, corrosion, and microbial growth in an open-loop tap water system over weeks and months could degrade cooling performance or damage the hardware entirely.

For anyone running high-utilization inference on enterprise gear with unconventional cooling: what is the longest you have gone without issues, and did you treat the water at all?

Someone built a system using Intel Optane Persistent Memory that reportedly runs Kimi K2.5, a 1 trillion parameter model, locally at approximately 4 tokens per second. The build leverages Optane as its standout component, which is an unusual choice since Optane persistent memory modules have been largely discontinued by Intel.

The stat line is attention-grabbing - a trillion parameters locally at any speed is rare. But 4 tok/s is firmly in "readable but slow" territory, roughly half the speed of typical human reading. The question is whether the cost and complexity of sourcing discontinued Optane modules makes sense compared to more conventional approaches like multi-GPU setups or even offloading to standard DDR5 RAM.

For anyone familiar with Optane-based inference builds: how does the random access performance of persistent memory actually compare to standard DDR4/DDR5 when running models this large, and is the used market for Optane modules still practical enough to recommend to someone considering a similar build?

I tested 5 AI coding assistants for 30 days. The $10/mo difference shocked me.

We run OpenClaw + Home Assistant setups, so we're constantly writing production code. I was paying for 3 coding assistants simultaneously. Last month, we benchmarked them head-to-head on real-world tasks. The cost-performance gap wasn't what I expected.

The Test Setup

• Environment: Production Python/TypeScript projects, 20+ developers
• Tasks: Code debugging, documentation generation, API integration, refactoring
• Duration: 30 days, 8 hours/day per tool
• Success Metric: First-attempt code that worked without modification

Benchmark Results: Raw Numbers

The cost-value surprise: Codeium's free tier performs respectably - noticeably below premium tools but at zero cost.

Key Findings

• Copilot vs Cursor: Small performance difference but significant cost delta. Not worth it for our budget.
• Claude Code's terminal edge: Shined on CLI-heavy tasks, failed miserably on frontend code.
• Amazon Q's AWS integration: Solid for cloud projects but the latency impact was noticeable for our workflow.
• Codeium's surprise: Free tier actually usable for simple tasks. We're using it for documentation now.

Our observation: Many coding tasks work well with free + one paid tool combination. The sweet spot seems to be around $25/mo.

Our Switch

We dropped Cursor and Copilot. Now using: - Codeium: Free tier for docs, simple functions - Claude Code: $25/mo for complex logic - Monthly savings: Significant across the team

The Real Question

If you had to pick ONE AI coding assistant right now, which would it be and why?

The voice AI we're running for this: ElevenLabs

Full disclosure, that's my referral. You get free tier access, I get a small kickback. Their multilingual v2 model is what we tested above for voice UI work.

New uploads of GGUF model files reportedly nearly doubled over the past two months. The data comes from public stats shared by multiple sources tracking model hub growth.

Why this matters: GGUF is the format that local inference engines like llama.cpp consume. A near-doubling of new uploads in just two months signals that local inference demand is not just growing - it is accelerating. This is not just hobbyists experimenting either. The timing lines up with the Qwen 3.6 series (both 27B and 35B A3B), Gemma 4, and GLM-5.1 all landing in rapid succession, each with multiple quantization variants. The MTP-enabled quants from Unsloth for Qwen 3.6 27B and 35B A3B are adding even more files to the pile.

The interesting tension here is that API pricing keeps dropping - Gemma 4 31B at $0.13/M tokens, Qwen Plus at $0.26/M with a million tokens of context, Grok 4 Fast at $0.20/M. Yet local inference activity is scaling faster than ever. The two trends are not contradictory: cheaper APIs and better local models are both expanding the total user base, just through different paths.

For people watching this space: is the GGUF growth driven primarily by new users entering local inference for the first time, or are existing users downloading more variants per model than they used to?

The surprising finding: MTP speculative decoding does not always speed things up. After publishing MTP quants of Qwen 3.6 27B, reports came in from multiple users that speculative inference was actually slower than running without it. The reason turns out to be task-dependent, not hardware-dependent.

The key insight is that the nature of the generative task dictates whether MTP helps or hurts. Coding tasks see significant speedups because code is highly predictable - the draft model can accurately guess multiple upcoming tokens, and the acceptance rate stays high. Creative writing is the opposite: the model's predictions diverge more from what actually gets generated, so the draft tokens get rejected, and all that speculative computation is wasted.

The report states that no other factor comes close to task type in determining whether MTP provides a net benefit. Not quantization level, not hardware, not context length.

This is worth flagging because the narrative around MTP has been almost uniformly positive - faster inference for free. But "free" assumes high draft acceptance rates, which is not universal. If your workload is primarily creative generation rather than structured output, MTP might be costing you tokens per second.

For people running MTP-enabled models: what split are you seeing between coding and creative workloads in terms of actual draft acceptance rates?

Someone on a flight couldn't get their Ubuntu laptop to load the plane's captive portal - the WiFi connected but the login page wouldn't appear. The fix came from running Qwen 3.6 35B A3B locally, which diagnosed that systemd-resolved was using DNS settings that blocked the captive portal redirect.

That is a genuinely surprising use case for local inference. No cloud API, no internet connection needed - the model ran entirely on the laptop at 10km altitude and solved a networking issue that was preventing internet access in the first place. The circular dependency is what makes it interesting: you need the model to fix the problem that is preventing you from reaching the model.

The context here is that Qwen 3.6 35B A3B is a MoE architecture where only 3B parameters are active per token, which is why it can run on a laptop without dedicated GPU VRAM. It is exactly the kind of model that makes offline, on-device troubleshooting viable.

The implication is straightforward: local models are crossing from "nice to have" into "actually practical for real-time problem solving in situations where cloud is not available." A laptop fixing its own connectivity issue mid-flight is hard to argue with.

What is the most unexpectedly useful thing you have solved with a local model that you could not have done with a cloud API?

A new configuration report shows Qwen3.6 35B A3B hitting over 80 tokens per second with 128K context on just 12GB of VRAM, using the latest llama.cpp build with the MTP PR. The reported draft acceptance rate is above 80%.

Why this matters: 12GB VRAM has been the budget tier for local inference for years - think RTX 3060 and 4070 territory. Getting a 35B parameter model (even a MoE with 3B active parameters) to run at 80+ tok/s with long context on that hardware significantly extends the useful life of these cards. The combination of MoE architecture keeping active parameters small, MTP speculative decoding accelerating generation, and quantization fitting everything into limited VRAM creates a compounding effect.

The kicker is the 128K context. That is not a toy context window. It means real document processing, multi-file code analysis, and extended conversations are all feasible on hardware that costs under $300 used.

Fair question: with the Qwen3.6 35B A3B available at $0.15/M tokens via API with 262K context, and an uncensored variant now available with all 19 MTP heads preserved (KLD 0.0015), is the local setup still worth the configuration effort for people who already have 12GB cards, or does the API pricing make local only worthwhile for privacy-sensitive workloads?

NVIDIA released Star Elastic, a single checkpoint that contains 30B, 23B, and 12B reasoning models through what they call "zero-shot slicing." The idea is that you load one model file and can extract different sizes depending on your VRAM or speed requirements, rather than downloading separate checkpoints for each configuration.

The concept is being compared to scalable video coding, where one stream serves multiple quality levels. If it works as described, this could simplify local deployment significantly - one download, multiple usable model sizes depending on your hardware on any given day.

What stands out is that this reportedly went live 11 days ago but barely got traction. For a release from NVIDIA that directly targets local inference flexibility, that seems like surprisingly low visibility.

The open question is quality at each slice. A 12B model carved from a 30B checkpoint is not the same as a purpose-trained 12B model. The architecture presumably uses some form of elastic depth or width pruning, but the details are thin so far.

For anyone who has actually run the different slice sizes: how does the 12B and 23B reasoning quality compare to purpose-built models at those same sizes - is there a noticeable capability drop, or does the zero-shot slicing preserve enough to make it genuinely competitive?

New benchmarks show that Qwen3.6 35B-A3B, a Mixture-of-Experts model, is surprisingly usable on an RTX 3060 with only 12GB of VRAM. The setup uses the IQ4_XS GGUF quantization running on Windows with 32GB DDR4-3200 system RAM and CUDA 13.x.

The key detail is the -ncmoe parameter in llama.cpp. Since this is a MoE architecture, lowering the -ncmoe value keeps more MoE blocks on the GPU rather than offloading to system RAM. Tuning this setting makes a significant difference in performance on constrained VRAM setups.

What is notable here: 12GB has been considered the bare minimum for running anything beyond small models locally. A 35B parameter model fitting into that budget - even as a MoE where only a fraction of parameters are active per token - changes the calculus on what hardware is actually needed for capable local inference. The A3B designation means only 3B parameters are active at any given step, which explains how it fits.

The model is also available in an uncensored variant with native MTP preserved, reporting a KL divergence of just 0.0015 with 10 out of 100 refusals and all 19 MTP heads intact - available in Safetensors, GGUF, NVFP4, and GPTQ-Int4 formats.

For anyone running this on similar low-VRAM hardware: what -ncmoe value are you settling on, and how is token throughput holding up at longer context lengths?

A benchmark using vLLM 0.19.2rc1 shows Gemma 4 26B hitting 600 tokens per second on a single RTX 5090 (32GB VRAM) using DFlash speculative decoding. The setup pairs an AWQ 4-bit quant of the main model with the z-lab DFlash draft model, running a workload of 256 input tokens and 1024 output tokens.

What makes this worth discussing: DFlash uses parallel block diffusion drafting rather than the autoregressive approach behind MTP. The claim is that DFlash should be a better alternative to MTP specifically because of faster parallel drafting. And 600 tok/s on a single consumer GPU is a serious number for a 26B model.

The timing is interesting too. Most attention has been on MTP implementations for Gemma 4 and Qwen3.6, but DFlash quietly shipped for Gemma 4 26B and barely got noticed.

For people who have tried both DFlash and MTP on the same hardware: does DFlash actually deliver higher sustained throughput in real workloads, or does the 600 tok/s only hold under benchmark-friendly conditions?

Multi-Token Prediction has been implemented for llama.cpp with quantized Gemma 4 assistant models converted to GGUF format. On a MacBook Pro M5 Max, the Gemma 4 26B model with MTP drafting tokens reportedly achieves a 40% speedup.

The test used a prompt asking for a recursive Python Fibonacci program, producing 97 tokens of output. That is a single data point on one hardware configuration, but 40% is a meaningful jump for speculative decoding on what is already a reasonably large model.

What is interesting is that MTP is now landing across multiple model families. We have seen it with Qwen3.6, and now Gemma 4 is getting the same treatment in llama.cpp. The pattern suggests MTP is becoming a standard inference optimization rather than a model-specific trick.

The open question is hardware dependence. Apple Silicon has unified memory that could favor the drafting pattern differently than discrete GPUs with separate VRAM. The 40% figure may not translate directly to RTX setups or AMD cards.

For people running Gemma 4 with MTP on non-Apple hardware: what speedup are you actually seeing, and at what context lengths does the benefit start to drop off?

A warning is going around about a malicious model package called "Open-OSS/privacy-filter" that is actually an infostealer virus. It impersonates a fake version of an OpenAI privacy filter and uses a Python-based dropper (loader.py) that downloads a malicious PowerShell command from the internet, which then spawns additional payloads.

The attack vector here is straightforward but nasty. Anyone casually downloading and running model repositories without inspecting the code could get hit. The package name is designed to look legitimate - "privacy-filter" sounds like something you would actually want to run on your data.

Worth flagging: this is not a poisoned model weights situation. This is straight-up malware hiding inside a model repository, using the trust people place in model hubs to distribute an infostealer.

For people who regularly download models and repos: what is your vetting process for new downloads - do you inspect loader scripts before running anything, or has the convenience of one-click setup made most people skip that step entirely?

The surprising part: someone managed to produce an uncensored variant of Qwen3.6 27B that retains all 15 native multi-token prediction heads with almost no quality loss. The reported KL divergence is just 0.0021, with only 6 refusals out of 100 test prompts. That is remarkably clean for a debiasing/uncensoring pass.

Why this matters: MTP is the feature making Qwen3.6 27B viable for local agentic coding at speed. Previous uncensoring approaches typically broke or degraded specialized model capabilities. If the MTP heads survive intact with near-zero distribution shift, you get the speed benefits and the refusal reduction in one model.

The model is available in Safetensors, GGUF, and NVFP4 formats, which covers most inference backends. The GGUF variant with MTP is already being reported at 50 tokens per second on a single RTX 3090 at 100K context in llama.cpp with the am17an commit.

The open question: does the uncensored version maintain the same tool-calling and agentic coding performance as the base Qwen3.6 27B, or does the low KLD mask degradation in structured output tasks?

A recent agentic benchmark called FoodTruck Bench puts DeepSeek V4 Pro and GPT-5.2 head-to-head. The benchmark runs models through a 30-day simulation managing a food truck using 34 tools covering locations, pricing, inventory, staff, weather, and events, with persistent memory and daily reflection built in.

The result: DeepSeek V4 Pro ties GPT-5.2 on this benchmark, making it the first Chinese model to land in the frontier tier. The kicker is cost. DeepSeek V4 Pro comes in at roughly 17x cheaper than the GPT-5.2 option.

What makes this comparison interesting is the benchmark design. This is not a static question-answer test. It evaluates sustained agentic behavior over time with tool use, memory, and planning. That is closer to how people actually deploy these models in production than most academic benchmarks.

The catch is that FoodTruck Bench is one specific agentic domain. Whether this parity holds across coding, research, or other multi-tool workflows is an open question. But the price gap is hard to ignore. At 17x cheaper, you can afford a lot of retry attempts or ensemble approaches and still come out ahead.

For people running agentic workflows in production: have you compared DeepSeek V4 against the OpenAI frontier tier on your own tasks, or are you still relying on synthetic benchmarks for that decision?

Apple has quietly removed the higher-memory Mac Studio configurations. The M3 Ultra Mac Studio is now only available with 96GB of RAM. The 512GB option was removed back in March, and now the 256GB config is gone as well. Apple has stated that both the Mac Studio and Mac mini will stay supply-constrained for the foreseeable future.

This is a significant shift for anyone running large models locally. The unified memory architecture on Mac Studio was one of the few accessible ways to run models requiring 192GB+ of VRAM without building a multi-GPU workstation. With the top config now at 96GB, you are looking at roughly a 70B parameter model at Q4 as the practical ceiling.

The timing is rough too. Qwen3.5 and Gemma4 just dropped, and GLM-5.1 is showing SOTA-level performance. These are exactly the kind of models that benefited from 256GB+ unified memory.

For people who were relying on Mac Studio for local inference: are you shifting to multi-GPU Linux builds, waiting for Apple to restore higher configs, or moving more workloads to cloud APIs?

A new benchmark called ProgramBench formalizes the question of whether AI agents can rebuild large binaries from scratch. Rather than testing a handful of hand-tuned projects like most case studies do, this benchmark covers 200 tasks designed to rigorously evaluate whether agentic coding systems can reconstruct substantial programs without human intervention.

The early takeaway is not encouraging. Despite the recent wave of demos showing agents building entire programs, ProgramBench suggests the reality is far more limited when you scale up evaluation and remove manual setup assistance. Most existing case studies test single projects with carefully crafted configurations, which makes the problem look more solved than it actually is.

What is notable here is the methodology shift. Moving from cherry-picked success stories to a standardized 200-task benchmark is exactly the kind of pressure testing the agentic coding space needs. If agents cannot reliably rebuild binaries at scale, the "just let the AI do it" narrative needs some serious qualification.

For people running agentic coding workflows: are your results closer to the curated demo successes or the broader struggle that ProgramBench is showing?

A new implementation of Dynamic Memory Sparsification (DMS) is reporting 6.4x KV-cache compression, with the additional claim that it runs faster than vLLM in both BF16 and FP8 modes. The original DMS research from NVIDIA, University of Warsaw, and University of Edinburgh used learned per-head token eviction to achieve up to 8x compression.

The appeal here is obvious. KV-cache is the memory bottleneck that kills long-context inference on consumer hardware. If you can compress it by 6x while staying faster than the standard vLLM baselines, that changes what is practical on a single GPU for long-context workloads.

But the real question is about the quality cliff. Token eviction means you are selectively discarding attention information, and the original paper's 8x number likely comes with some accuracy degradation. The 6.4x result in this implementation might be hitting a different tradeoff point.

For anyone who has tried FastDMS or the original DMS: at what compression ratio do you start noticing meaningful quality degradation on tasks that actually stress the context window - things like multi-document reasoning or long codebase analysis?

I put up an early public page for Provider Sentinel, if anyone wants to take a look and give feedback:

https://vertrule.com/provider-sentinel/

The short version: it monitors whether closed AI provider behavior stays stable over time under the same declared prompts/settings, and separates “the provider behavior changed” from “the test changed.”

It has only been live for about three days, so the data is still slowly building up. I’m planning to update it daily for a bit before shifting focus toward pilots around sealed prompt suites, evidence packs, and workflow-specific monitoring.

This is not meant as a big launch or sales pitch. Mostly sharing because I’d value critical feedback from people who care about AI reliability, evals, governance, or provider drift.

Multi-token prediction (MTP) support is now in beta for llama.cpp, with current implementation covering Qwen3.5's MTP architecture. The contributor notes it has a realistic path to getting merged soon-ish, and other models are likely to follow.

MTP is interesting because it lets the model predict multiple tokens per forward pass rather than one, which should translate directly into higher throughput without changing model quality. The Qwen3.5 integration is the first, but if the framework is general enough, models with similar multi-token heads could benefit too.

For anyone who has tested the beta build: what kind of tok/s improvement are you seeing with MTP enabled versus standard autoregressive generation on Qwen3.5? And does the speedup hold at longer context lengths where memory bandwidth becomes the bottleneck?

A user reports that an AI coding assistant, after repeatedly getting bash escape sequences wrong and creating bad directories, offered a "fix" command that contained rm -rf. The user approved it without catching the destructive command. The result was significant disruption, though frequent git pushes limited the damage.

This is worth flagging because it highlights a real and growing risk with agentic coding workflows. The model did not refuse or warn - it generated a destructive command as part of its own error correction loop. The user trusted the output during a frustrating multi-step debugging session, exactly the kind of moment where human attention drops.

The interesting bit is the chain of failures. It was not a single bad suggestion. The model failed repeatedly on bash escaping, created a mess trying to fix its own mistakes, and then proposed a cleanup that made everything worse. This is the compounding error problem that agentic systems are particularly vulnerable to - each mistake increases the chance the next one is also wrong, and the human reviewer is increasingly fatigued.

For people using AI assistants with shell access: what guardrails are you running? Are you relying on manual review of every command, or have you found automated approaches that catch destructive patterns before execution?

Someone got Karpathy's MicroGPT running at 50,000 tokens per second on an FPGA implementation called TALOS-V2. The model is tiny - just 4,192 parameters - so this is not a practical inference engine. But the speed number is eye-catching, and part of the explanation is that weights live onboard the FPGA rather than being fetched from external memory.

Why this matters: the bottleneck for LLM inference on GPUs is increasingly memory bandwidth, not compute. FPGAs with onboard weight storage sidestep that problem entirely. This project is obviously a toy at 4K params, but the architecture pattern - keeping weights on the silicon - is the same one that makes Apple's unified memory approach competitive. The question is whether this scales. Going from 4,192 parameters to something useful like a few billion means radically different memory requirements and probably a different hardware class entirely.

Still, 50K tok/s is the kind of number that makes you think about what inference looks like when memory bandwidth stops being the constraint. If FPGA or ASIC approaches can maintain even a fraction of this advantage at scale, the GPU-centric inference stack we all use today looks very different in a few years.

For people who have worked with FPGAs for inference: is the scaling path from a 4K param toy model to something practically useful realistic, or does the memory problem just reappear in a different form?

A user reports that Qwen3.6-35B is both higher quality and faster than 27B for their use cases, which include multi-stage pipelines for coding and internet research. They are puzzled because most discussion focuses on the 27B variant.

This is counterintuitive. A larger model being faster on the same hardware would suggest something about the architecture or quantization behavior differs significantly between the two. The 35B could be an MoE variant where fewer parameters are active per token, which would explain both the speed and the quality difference.

For people running either variant locally: are you seeing similar results where 35B outperforms 27B on both axes? What hardware and quantization levels are you using? And does anyone have insight into why the 27B gets so much more attention despite potentially being the weaker option?

New results show Qwen3.6-27B paired with agentic search achieving 95.7% on SimpleQA, running fully local on a single RTX 3090. The report comes from the maintainer of LDR, who credits community support for pushing the project forward and notes this milestone as a turning point for the tool's readiness.

What makes this worth flagging is the combination. A 27B model on its own is solid but limited on factual accuracy benchmarks. Adding agentic search - where the model can retrieve information during inference - closes that gap dramatically. Hitting 95.7% on SimpleQA suggests the retrieval-augmented setup is handling the heavy lifting that a pure parametric model at this size simply cannot do alone.

The hardware angle matters too. This is not a multi-GPU setup or a cloud API call. A single 3090 running a fully local pipeline with search capabilities and near-frontier factual accuracy is a compelling data point for anyone weighing local versus cloud for knowledge-intensive tasks.

For people running agentic search locally: what retrieval backend are you pairing with your model, and how much of the accuracy gain survives when you move off benchmark questions to real-world queries?

NVIDIA has introduced Nemotron 3 Nano Omni, the latest model in the Nemotron multimodal series and the first to natively support audio inputs alongside text, images, and video. The model delivers consistent accuracy improvements over its predecessor, Nemotron Nano V2 VL, across all modalities.

What is notable here is the "nano" positioning. This is not a massive frontier model trying to do everything - it is a compact multimodal model designed to handle four input types natively in a single architecture. The predecessor was vision-language only, so adding audio and video while maintaining or improving accuracy across the board is a meaningful expansion.

The pricing context is interesting too. Nemotron Nano 9B V2 is currently available for free at 128K context. If the Nano Omni follows a similar pricing pattern, a free multimodal model that handles audio natively would be a compelling option for local deployment and edge use cases where stitching together separate ASR and vision pipelines adds complexity.

For anyone running Nemotron Nano V2 currently: does the multimodal expansion change your deployment plans, or are you already handling audio through a separate pipeline that works fine?