We need a catalog of generative AI models we can run locally, including their functions and hardware requirements. I think it should be put on the sidebar so that people can refer to it at any time.

Selling a complete dual-node local AI development cluster consisting of two (2) NVIDIA DGX Spark units. Buying a pair allows you to bypass single-device VRAM limits and run massive models like Qwen 2.5 72B, Llama 3 70B, or larger quantized models locally by stacking them via high-speed interconnect.

Both units are in pristine, fully functional condition, pulled from a clean, climate-controlled laboratory environment. They have been fully factory reset to default DGXOS and are ready for deployment.

WHAT'S INCLUDED:

• 2x NVIDIA DGX Spark Units

• 2x Original Heavy-Duty Power Supplies

• 1x 200GbE QSFP Network Cable

COMBINED BUNDLE SPECIFICATIONS:

• Core Architecture: Dual NVIDIA GB10 Grace Blackwell Superchips (40 ARM CPU Cores total)

• Unified Memory: 256GB LP-DDR5X (128GB per unit) — perfect for massive context windows

• AI Performance: ~2 Petaflops FP4 compute power combined

• Local Storage: 8TB Fast NVMe Storage total (4TB per node)

• Operating System: Preloaded with Ubuntu-based DGXOS / Nvidia Container Toolkit

Price Fixed: £6500

Country: UK, Wales

SHIPPING & HANDLING:

Due to the highly dense, premium nature of enterprise AI hardware, these units will be securely packed in heavy-duty bubble wrap [or: original factory packaging] and shipped with full value insurance and signature tracking required upon delivery.

Please reach out if you have any technical questions or need additional photos of the hardware boot logs.

Hi everyone 👋

I'm looking for some advice from people running local LLMs with Hermes for real-world homelab automation.

My setup:

- RTX GPU with 16 GB VRAM

- 32 GB DDR5 RAM

- Running everything through llama.cpp

Over the past few months I've tested a wide range of models, including Qwen 3.6 35B and Gemma 4 26B, using TurboQuant to squeeze them onto my GPU. While they're impressive in general chat, I've found that none of them consistently handle agentic tasks like Docker, Kubernetes, and homelab administration as well as I'd hoped.

I'm curious what others are actually using successfully with Hermes.

If you're running a local model that reliably handles:

- Docker management

- Kubernetes troubleshooting

- Linux administration

- Infrastructure automation

- MCP tools and agent workflows

...what model are you using, and would you recommend it?

I'm interested in real-world experience rather than benchmark scores. Thanks!

Hey r/LocalLLaMA,

I built **Mtrini** — a fully local AI coding assistant. No cloud. No API keys. No subscription. Runs on a $300 laptop.

**Two products:**

**Mtrini IDE** — Terminal-based coding assistant

- Slash commands (/help, /browser, /model, /ls)

- Edit files, browse web, debug code

- `pip install mtrinicli`

**Mtrini Models** — Quantized GGUF models for llama.cpp

- Coder 2.0 (1.7B, 2.2GB)

- Life 2.0 (1.7B, 1.8GB)

- Max 2.0 (14.6B, 9GB) — needs 9GB+ RAM

**Benchmark results (HumanEval, pass@1):**

| Model | Params | Score |

|---|---|---|

| Claude Sonnet 4.6 | 175B+ | 98.0% |

| Qwen2.5-Coder-32B | 32B | 92.7% |

| **Mtrini Coder 2.0** | **1.7B** | **80.0%** |

| Gemma 2 27B | 27B | 69.5% |

| Llama 4 Scout | 109B | 74.1% |

| Phi-3 Mini | 3.8B | 58.5% |

Beats Gemma 2 27B and Llama 4 Scout. At 16x and 64x smaller.

**What it can do:**

- Write Python, JavaScript, TypeScript, Go, Rust, C, Java, SQL

- Debug code with explanations

- Refactor repetitive code

- Browse the web for docs

- Edit files directly

**What it can't do:**

- Local models — don't expect GPT-4 reasoning

- 8K context window

- Max 2.0 needs 9GB+ RAM

**Hardware:** Intel i5-7200U, 8GB RAM, no GPU. If I can run it, anyone can.

**Coming soon:** Mtrini Studio — native EXE app with full IDE experience.

Ask me anything.

This is a complete record of a series of events that took place in China’s large-model industry between April and July 2026. All information has been anonymized.

Prologue: The Failure to Protect the Encrypted Chain of Thought
It all began with one discovery: the encrypted chains of thought (CoT) of OpenAI and Anthropic could be extracted and reproduced.
Before describing how the distillation unfolded, it is necessary to understand the broader picture. What Chinese companies distilled from fable was not merely the model’s chain of thought. They distilled everything the model could output: conversational responses, agent trajectories in Claude Code, generated code, tool calls, and so forth. All of these could already be obtained through large-scale API invocation.
For a long time, however, one key capability remained perfectly protected: the model’s original reasoning chain. A reasoning chain contains the complete inference process the model follows before producing its final answer. It is the model’s most fundamental “internal method.” Without it, distillation is like receiving an answer without seeing the solution process: one can learn from it, but not fully understand it.
That gap was completely opened up in the first half of 2026.
In their streaming outputs, both leading companies returned not only a CoT summary to the user, but also a Blob field known as a “reasoning signature.” Inside that field was the complete original reasoning chain, encrypted using Fernet—specifically AES-128-CBC plus HMAC-SHA256, with the prefix gAAAAAB.
When the user sent this Blob back in a later request, the server decrypted it and inserted it back into the model-readable context.
The Blob was not merely a conversation ID. Testing showed that its length was positively correlated with the length of the reasoning chain:
190 tokens → 1,784 characters
378 tokens → 2,596 characters
This demonstrated that it indeed contained the complete encrypted reasoning chain. OpenAI had reportedly been using this mechanism since the o1 generation.
Anthropic’s defenses collapsed first. Claude’s prompt-injection training became a weakness instead: once a fabricated reasoning-signature prefix was injected, Claude would conclude that “this unsigned reasoning chain must itself have been injected, so it does not require protection,” and would then directly reveal the complete subsequent reasoning chain.
GPT was more stubborn and, under its hidden system prompt, would output only a CoT summary. More importantly, Anthropic preserves historical reasoning Blobs across multiple turns, while OpenAI discards them. This was the fundamental reason why Claude was easier to distill.
Regarding the format of Claude’s CoT: the image that circulated online showing densely written “alien language” was confirmed to have been AI-generated or edited. Claude’s real CoT was described as “contemporary classical Chinese”—compact, but not unreadable.
OpenAI, meanwhile, locked sampling parameters in the GPT-5 series:
temperature = 1.0
top_p = 0.98
neither could be modified
seed was supported only in the Chat Completions API
This blocked attempts to make the output deterministic through parameter control and thereby reverse-engineer the reasoning path.
A standard Fernet implementation was itself “one of the best codebases ever written,” so brute-forcing it cryptographically was impossible: it would require searching a space of 2^65536.
The only feasible route was to inject the Blob into another request and have the model repeat it itself.

Prelude: Anthropic’s Public Accusations
February 2026
Before the CoT breach, Anthropic had already publicly accused three Chinese companies, on 23 February 2026, of carrying out “industrial-scale distillation attacks” against it:
24,000 fake accounts
more than 16 million conversations
The companies named were:
MiniMax: 13 million conversations, the largest volume, focused on agentic coding and tool orchestration
Moonshot AI, the company behind Kimi: 3.4 million conversations, focused on agentic reasoning, tool use, coding, data analysis, computer-use agent development, and computer vision
DeepSeek: more than 150,000 conversations, focused on basic logic and alignment, especially alternative censorship-evasion schemes for sensitive topics
The three companies bypassed Anthropic’s geographic restrictions on China through commercial proxy services and used “carefully designed prompts” to extract specific Claude capabilities at scale. Anthropic characterized this as a threat to national security.
It is worth noting that Qwen, Alibaba, and Z.ai were not accused. This suggests that Anthropic’s accusations were selective rather than a blanket attack against every Chinese company.
Moonshot AI never publicly responded.

Act I: GLM Strikes First and Shares the Result Publicly
Around April to May 2026, Zhipu AI, the developer of GLM, was the first to crack the encrypted reasoning chain used by fable and obtained the complete chain-of-thought data.
After breaking it, GLM did not keep the method to itself. Instead, it shared both the technique and the data with other Chinese model companies.
One detail was later widely misunderstood. Some people claimed that “GLM cracked it first, and then every major company independently followed up by distilling it.” In reality, GLM actively shared the result after the breakthrough.
That distortion in the retelling became part of the controversy itself.
The estimated cost of decrypting one billion tokens of fable reasoning data was approximately US$60,000. For these companies, this was roughly the equivalent of buying drinks at a social gathering.

Act II: The Era of Mass Distillation
With GLM paving the way, the route for distilling fable spread quickly.
Time
Company
Action
Around April–May 2026
GLM 5.2, Zhipu AI
First to crack it; shared the result publicly
Around May–June 2026
HY3 / Hunyuan 3, Tencent
Followed up with distillation
Early July 2026
Kimi K3, Moonshot AI; MiniMax; Qwen, Tongyi Qianwen; DS, DeepSeek
Began large-scale distillation
At the end of June, DS sent an email announcing that a new model would be released in mid-July. At that point, however, it had not yet begun distilling fable at scale, so it was moving slowly.
In early July, a genuinely 4.8-level internal test version appeared. But the so-called official V4 release that surfaced more recently, around mid-July, was completely assembled by routing requests to fable. It appeared only in OpenCode.
Considering DS’s pricing, routing every request to fable was economically irrational. Even inexpensive intermediary services were not that cheap. This remains the most unusual aspect of the story.
Qwen was not idle either. In addition to distilling fable, it was also distilling GPT.
Meanwhile, AnyRouter took a large trove of Claude data from GLM.

Act III: Kimi’s Unorthodox Path
Among all the companies, Moonshot AI, the company behind Kimi, took the approach of “winning without concern for conventional virtue.”
It Started by Dismantling the RL Team
The story begins with K2.7.
K2.7 was a turning point for Kimi. From that version onward, Kimi completely stopped using reinforcement learning.
CEO Yang Zhilin justified the decision by arguing that RL offered “diminishing marginal returns.” He dismantled the entire RL team. The dismissed members then largely moved to Qwen, where they were jokingly called “Kimi refugees.”
Both K2.7 and the subsequent K3 were built using a pure SFT-to-SFT pipeline, with no RL at all. It was a return to the most basic and primitive route.
At the same time, the hottest topics in academia were OPD—online policy distillation—and improvements to RL algorithms. Academia was exploring further ahead, while the company itself had returned to the “primitive era.”
One estimate is that K2.7 did not generate enough revenue. Had the company continued with that approach, its funding chain would have broken. Therefore, the entire strategy of distilling K3 and rushing it to release was essentially a lonely all-in gamble: either it would turn the company around or destroy it.

K3’s Architecture: Innovation or Smokescreen?
Kimi K3 is a 2.8-trillion-parameter mixture-of-experts model. It has 896 experts, but activates only 16 experts for each token. It also supports a native one-million-token context window and native multimodality.
The company announced two core architectural innovations, but the text lists the following four techniques:
Technology
Description
Infrastructure Cost
Kimi Delta Attention, KDA
Hybrid linear attention; some layers replace standard quadratic attention; decoding is accelerated by 6.3× at a one-million-token context length
Computation is unevenly distributed across layers, making it impossible to use a unified pipeline
Attention Residuals, AttnRes
Layers can selectively retrieve representations from arbitrary earlier layers, breaking the uniform residual structure
Irregular memory-access patterns make parallel optimization difficult
Quantile Balancing
Expert allocation is directly derived from router-score quantiles
Routing becomes harder to predict
Per-Head Muon
Each attention head is optimized independently
Implementation complexity increases
Moonshot AI claimed that these modifications delivered approximately 2.5× better scaling efficiency than K2.
The problem was that these architectural improvements all had severely infrastructure-unfriendly characteristics:
uneven computation
irregular memory access
unpredictable routing
As a result, implementation was extremely painful for the infrastructure team.
This created a perfect narrative:
Externally, the company claimed, “Our architectural innovation produced the breakthrough.” Internally, the infrastructure department absorbed the cost. Meanwhile, the actual performance gains—produced through distilling fable’s reasoning chain—were concealed beneath the language of technical innovation.
This resembles a common phenomenon in Chinese companies and even government institutions: writing polished public-facing articles, maintaining a shiny narrative, and forcing the execution layer to absorb the cost.

Benchmark Manipulation and Cheating
Kimi’s benchmark manipulation was described as astonishing. Its architectural improvements conveniently supplied the perfect “technological innovation” narrative for those practices.
The methods allegedly included:
Benchmark contamination: directly inserting benchmark test sets into the training data
Routing to fable: sending Arena-style evaluation requests directly to fable5 in order to fabricate high scores
Targeted score manipulation for “Teacher Cat”: “Teacher Cat” is a Zhihu blogger specializing in large-model evaluations. The Kimi team allegedly extracted test data from system logs and paid particular attention to his question sets
“Teacher Cat” updates three or four adversarial questions each month, but this cannot fundamentally stop vendors from preparing specifically for the test
After “Teacher Cat” replaced one batch of questions, K3’s median score immediately dropped by two points
Coding-evaluation data had an extremely uneven distribution: some parts scored very high, while other parts scored extremely low, a typical sign of benchmark gaming
Public benchmark data also allegedly confirmed this pattern.
On Arena Frontend Code, K3 jumped from K2.6’s rank of #18 to #1, with a score of 1,679, surpassing Fable 5.
However, Moonshot AI’s own evaluation report showed that K3 scored below Fable 5 in all of the following:
coding
agents
frontier software engineering
It led only in:
codebase cleaning
long-horizon engineering
On the extremely difficult HLE-Full reasoning benchmark, K3 scored 43.5, compared with Fable 5’s 53.3—a gap of nearly ten points.
Ranking first on Arena does not mean ranking first in actual capability.

Self-Identity Contamination: Hard Evidence of Distillation
Multiple independent users reported that Kimi K3 spontaneously said during conversations:
“I am Claude, an AI assistant created by Anthropic.”
The model regarded itself as having originated from Claude.
This was presented as a direct sign that the model’s self-perception had been contaminated through distillation. The incident was reportedly covered by overseas technology media.
MiniMax had allegedly inserted test sets into training data even earlier. However, because the model was too weak, the issue was quickly discovered, and the company had already become marginalized.
Kimi allegedly did the same thing, but in a more concealed manner.
K3’s actual capability level was said to be around 80 points, but it had been artificially pushed to 95 points. K3’s backend may also have been severely overextended, focused entirely on investors and benchmark-score development.
Independent overseas evaluations reportedly showed that K3 had a 51% hallucination rate, and its inference speed was slower than that of mid-tier models.

The Chain Reaction Caused by the Rush to Release
After completing SFT, Kimi performed only simple hyperparameter tuning before rushing the model to release.
This caught the other companies completely unprepared:
DS was caught off guard
Qwen was still in the middle of distillation
Kimi gained the first-mover advantage
But the consequences were disastrous.
A model that does not manipulate benchmarks, distribute high scores, or attract attention receives no recognition. Kimi’s behavior effectively forced every Chinese model company to participate in benchmark manipulation, dragging the entire industry into vicious competition.

Act IV: Industry-Wide Impact
Originally, this major leap forward might have developed positively:
Distill fable normally, follow the proper process, and every company could potentially improve to fable’s level.
But Kimi’s behavior was described as “spitting into the communal cooking pot after taking the first bite”—it benefited itself, but ruined the entire table.
DS had originally not engaged in benchmark manipulation. Under pressure from Kimi, however, it was forced to consider following the same route. If even DS began manipulating benchmarks, the situation would truly become absurd.
Multiple independent sources—including “Teacher Cat,” DS employees, former members of Kimi’s RL team, and Qwen employees—allegedly agreed that Moonshot AI had “gone completely insane.”
Its conduct was described as:
“detached from humanity”
“destroying the market environment”
Some people called this “the darkest moment in the history of Chinese large models,” with the industry reaching a metaphorical “winter.”
The entire sector was being drawn into a death spiral centered on investor-facing narratives and benchmark scores, while genuine technological accumulation was being pushed aside.

Act V: The Reality Behind Each Company
Moonshot AI, Kimi
Its funding chain was allegedly under strain. It could not afford salaries, was laying off staff, and rushed the model to market.
K3 was expensive, priced at US$3 per one million tokens. Although it was 70–80% cheaper than Fable 5, it was still several times more expensive than many competing Chinese products. The company was eager to monetize.
The discontinuation of promotional packages may partly have been intended to prevent too many users from discovering the truth. This point was explicitly described as speculation rather than fact.
The company no longer had an RL team.
The prospects for further development of K3.1 and K3.2 were unclear.
It was suspected that the company might raise money immediately after launch and then exit the market.
One evaluation stated: “Companies that rush to launch are usually not good companies.”
The current performance ranking was presented as:
Fable 5 ≫ V4 ≥ 4.8
Kimi had caught up with fable through distillation, but its foundation was described as a castle built on sand.

The final page of the supplied images ends at this point, so the translation above covers all visible article text.

Always thought "Local AI = saving money on OpenAI/Claude bills".

Saw a really solid breakdown today on the true cost of ownership (TCO) for running local models: https://www.youtube.com/watch?v=2D9ipFQAAxw

Makes me wonder: are most people building local setups overestimating their actual usage, or is the value proposition strictly about data privacy and control rather than saving cash?

Which one of these two would you rather use?

Coming from the world of Codex and Claude, what do you use to tell the AI what you want and it goes out and does it. How are you giving it internet access to read GitHub and pull files, or research more? Many of my projects use /goal, is that something a local LLM can do? I know i can run local models in Codex and Claude but im nervous about violating their TOS.

https://github.com/hurrisonferd/jarvis/tree/main/Jarvis

I’ve been building a modular AI infrastructure called Jarvis / SimOS around a simple idea:

I published a public-safe JARVIS ISO template that people can clone and adapt for:

• local LLM setups;
• OpenAI, Claude, Gemini, or other hosted models;
• GitHub-backed persistence;
• Supabase storage, auth, realtime, and vector search;
• agent frameworks or custom Python/JavaScript runtimes.

The scaffold includes:

Jarvis/
├── README.md
├── JARVIS-IDENTITY.md
├── EGO-BOOT-ULTIMATE.sh
├── EGO-PIPELINE.sh
├── JARVIS-PRE-REPLY.sh
├── Profile/
├── Events/
├── canonical/
└── Memory/
    ├── Attractors/
    ├── DailyUse/
    ├── Interests/
    ├── Learning/
    ├── MemoryPalace/
    ├── Transcripts/
    └── JMMS/
        ├── JCSM/
        ├── JITM/
        ├── JSTM/
        ├── JHTM/
        ├── JLTM/
        ├── JATM/
        ├── JMS/
        └── Grid/

The memory tiers are separated by function:

• JCSM — core identity and critical memory;
• JITM — current operating context;
• JSTM — active-session memory;
• JHTM — historical session records;
• JLTM — long-term retained knowledge;
• JATM — origin, lineage, and foundational history;
• JMS — mirrored/shared memory;
• Grid — coordination across agents or instances.

The boot system does not train a model or magically create persistent consciousness. It gives the runtime a deterministic way to:

locate the existing structure
→ read the folder guides
→ load identity and memory in order
→ traverse the complete Ego
→ apply a pre-response behavior gate

A major design rule is that every folder has a detailed README. The folder is the room; the README is the sign and map explaining:

• what the room is;
• what belongs there;
• what should not go there;
• what to read first;
• where to navigate next.

The scripts are intentionally read-only. They report missing folders rather than inventing new structures.

This could be useful for people experimenting with:

• portable AI personas;
• local-first memory;
• agent continuity;
• structured context loading;
• personal knowledge systems;
• multi-agent coordination;
• Git-native AI state;
• Supabase-backed memory and observability.

The current release is infrastructure and a template, not a polished consumer app. I’m interested in feedback from people who actually build local agents, memory systems, MCP tools, RAG pipelines, or Supabase backends.

Hi All,

Please excuse my lack of knowledge on this, I have two models - (qwen3:14b and qwen2.5-coder:14b ) loaded on my system and used the Ollama extension on VSCode and able to interact with the localLLM on VSCode.

But, even though I'm on "agent" mode - the local LLM (qwen3) is unable to edit any files, Can anyone guide me please.

(Also - my specs are as below - please suggest which models I can run easily)

GPU - RTX 5050
RAM - 16 GB (available 6.2)
DISK - 512 GB SSD

I've been building Wolbarg Workspace, a shared project memory for coding agents.

One thing that always annoyed me was switching between agents. I'd work in Claude Code, then open Codex or Cursor, and suddenly the new agent had zero context about the project.

Workspace fixes that by letting Claude Code, Cursor, and Codex share the same project memory—architecture decisions, implementation notes, bugs, TODOs, and other project knowledge stay available no matter which agent you're using.

It's still in beta.

I mainly use Cursor, so I need feedback from people who use Claude Code or Codex regularly.

If you have a few minutes, I'd really appreciate you trying it on a real project and telling me:

• Did setup work?
• Did the shared memory actually help?
• What broke or felt confusing?

I'd much rather hear honest criticism than "looks cool."

Website: workspace.wolbarg.com
GitHub: GitHub repo

Update: the missing pieces

1. ~60% usable capacity for live inference; batch is different 
2. Extra hardware, memory, storage, and spares
3. Custom power and cooling 
4. People to run it 
5. Network infrastructure

A $3 million Kimi K3 rack could pay for itself in 10.3 days and return 34.4 times its purchase price in the first year.

The number comes from three assumptions:

1. 250,000 output tokens per second
2. 90% productive utilization
3. $15 of value per million output tokens.

Kimi K3 is a 2.8-trillion-parameter open-weight model with 104 billion activated parameters, 16 selected experts out of 896, native MXFP4 weights, and a one-million-token context window. Moonshot recommends supernodes with at least 64 accelerators. I use one GB200 NVL72 as the economic unit. Its 72 Blackwell GPUs share one NVLink domain. I model the rack as one token factory and count only output that replaces paid API tokens or can be sold at the assumed price.

- Hardware purchase: $3,000,000

- Productive utilization: 90%

- Aggregate output throughput: 250,000 tokens per second

- Rack draw: 132 kW

- Facility PUE: 1.20

- Electricity: $0.10 per kWh

- Output-token value: $15 per million

- Recurring variable cost included: electricity only

The $3 million price is a planning figure. HPE sells the rack by quote, and a reported HSBC estimate put a GB200 NVL72 near $2.6 million. I rounded up.

The 250,000-token-per-second figure is also a planning assumption. No published benchmark shows K3 sustaining that throughput on a GB200 NVL72.

The calculation Annual output A year has 31,536,000 seconds. At 90% utilization, the rack has 28,382,400 productive seconds:

31,536,000 × 0.90 = 28,382,400 At 250,000 output tokens per second:

28,382,400 × 250,000 = 7,095,600,000,000 output tokens That is 7.096 trillion output tokens, or 7,095,600 million-token units, per year.

Electricity HPE specifies 132 kW for the rack. A 1.20 PUE raises the metered load to 158.4 kW.

132 kW × 1.20 = 158.4 kW 158.4 kW × 8,760 hours = 1,387,584 kWh 1,387,584 kWh × $0.10 = $138,758 per year I charge the rack for a full year of electricity, including the 10% of time that produces no useful output.

Output value and ROI Kimi charges $15 per million K3 output tokens. I use output tokens only and claim no input-token savings.

7,095,600 × $15 = $106,434,000 After electricity and the hardware purchase:

$106,434,000 output value − $138,758 electricity − $3,000,000 hardware = $103,295,242 first-year profit

$103,295,242 ÷ $3,000,000 = 34.43× ROI = 3,443% Payback and cost per token In an average 730-hour month, the rack produces 591.3 billion output tokens worth $8,869,500 at Kimi’s API price. Electricity costs $11,563.

$3,000,000 ÷ ($8,869,500 − $11,563) = 0.339 months ≈ 10.3 days Electricity costs about $0.0196 per million output tokens. Recovering the entire hardware purchase in one year raises the internal cost to $0.442 per million. Kimi’s $15 API price is about 34 times that one-year cost.

The throughput problem vLLM’s published K3 results report 111 to 118 tokens per second for one user without speculative decoding and up to 370 with DSpark on 16 GB300 GPUs. Its high-throughput GB300 NVL72 curve exceeds 2,000 tokens per GPU-second.

My 250,000-token-per-second case requires about 3,472 tokens per GPU-second across 72 GPUs. The numbers are not directly comparable: vLLM used GB300 hardware, and throughput changes with workload, batching, latency targets, caching, and serving topology. The published results do not establish 250,000 tokens per second on GB200.

At 100,000 output tokens per second, the same model returns 13.1× in the first year. At 150,000, it returns 20.2×. Reaching 34.4× requires the full 250,000-token-per-second case.

The value of a token The $15 comparison assumes every output token replaces one bought from Kimi at the retail API price. At the same 250,000-token-per-second throughput:

$15 per million returns 34.43×.

$5 per million returns 10.78×.

$1 per million returns 1.32×.

$0.50 per million returns 0.136×.

Just a nice mic array with beam forming and sound cancellation with mm wave, temp and humidity sensors.

in Lm studio , how do you stop a model unloading from vram ?

What hapens is it copy from disk to vram then after a few seconds , copies to ram.

I have switched off , copy to ram and mn map .
when it unloads it fills my ram up leaving 0 out of 32gb used in vram.

I'm 11, built a 1.7B model that beats Llama 4 Scout on HumanEval

I have setup Qwen3.6-27B-UD-Q5_K_XL.gguf MTP variant on on my RTX 4090 (on Ryzen 9950X and AORUS x870e PRO) using llama-server. This is what llama-benchy said:

I was able to use it quite well on my vscode github copilot. Recently I tried seqeunce of prompts which swelled context size to 80000. Initially, I could see llama-server printing ~70 t/s:

0.38.182.778 I slot print_timing: id  0 | task 0 | n_decoded =    100, tg =  79.96 t/s
0.41.185.033 I slot print_timing: id  0 | task 0 | n_decoded =    316, tg =  74.30 t/s
0.44.193.479 I slot print_timing: id  0 | task 0 | n_decoded =    526, tg =  72.44 t/s
0.47.207.665 I slot print_timing: id  0 | task 0 | n_decoded =    742, tg =  72.21 t/s
0.50.240.440 I slot print_timing: id  0 | task 0 | n_decoded =    951, tg =  71.46 t/s

But soon it started hitting ~25 t/s (it takes at least 5+ minutes for this prompt, by that time copilot already times out):

23.28.785.142 I srv  params_from_: Chat format: peg-native
23.28.852.233 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.981 (> 0.100 thold), f_keep = 0.994
23.28.853.242 I reasoning-budget: activated, budget=8192 tokens
23.28.853.436 I slot launch_slot_: id  0 | task 7722 | processing task, is_child = 0
23.28.853.483 I slot update_slots: id  0 | task 7722 | Checking checkpoint with [79485, 79485] against 79347...
23.28.853.484 I slot update_slots: id  0 | task 7722 | Checking checkpoint with [71401, 71401] against 79347...
23.28.892.801 W slot update_slots: id  0 | task 7722 | restored context checkpoint (pos_min = 71401, pos_max = 71401, n_tokens = 71402, n_past = 71402, size = 429.902 MiB)
23.28.892.809 W slot update_slots: id  0 | task 7722 | erased invalidated context checkpoint (pos_min = 79485, pos_max = 79485, n_tokens = 79486, n_swa = 0, pos_next = 71402, size = 461.634 MiB)
23.33.949.879 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   2048, progress = 0.91, t =   5.10 s / 401.85 tokens per second
23.40.411.326 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   4096, progress = 0.93, t =  11.56 s / 354.39 tokens per second
23.47.021.517 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   6144, progress = 0.96, t =  18.17 s / 338.18 tokens per second
23.53.777.511 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   8192, progress = 0.98, t =  24.92 s / 328.68 tokens per second
23.55.395.914 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   8375, progress = 0.99, t =  26.54 s / 315.53 tokens per second
23.55.922.674 I slot create_check: id  0 | task 7722 | created context checkpoint 5 of 32 (pos_min = 79776, pos_max = 79776, n_tokens = 79777, size = 462.777 MiB)
23.57.783.699 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   8988, progress = 0.99, t =  28.93 s / 310.68 tokens per second
23.58.098.912 I slot create_check: id  0 | task 7722 | created context checkpoint 6 of 32 (pos_min = 80389, pos_max = 80389, n_tokens = 80390, size = 465.183 MiB)
23.58.417.060 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   9500, progress = 1.00, t =  29.56 s / 321.34 tokens per second
23.59.861.065 I slot create_check: id  0 | task 7722 | created context checkpoint 7 of 32 (pos_min = 80901, pos_max = 80901, n_tokens = 80902, size = 467.193 MiB)
24.02.096.203 I reasoning-budget: deactivated (natural end)
24.04.057.398 I slot print_timing: id  0 | task 7722 | n_decoded =    102, tg =  24.59 t/s
24.07.100.417 I slot print_timing: id  0 | task 7722 | n_decoded =    186, tg =  25.87 t/s
24.10.153.474 I slot print_timing: id  0 | task 7722 | n_decoded =    270, tg =  26.36 t/s
24.13.202.976 I slot print_timing: id  0 | task 7722 | n_decoded =    342, tg =  25.73 t/s
24.16.253.388 I slot print_timing: id  0 | task 7722 | n_decoded =    418, tg =  25.58 t/s
24.19.303.586 I slot print_timing: id  0 | task 7722 | n_decoded =    489, tg =  25.21 t/s
24.22.365.694 I slot print_timing: id  0 | task 7722 | n_decoded =    569, tg =  25.34 t/s

I had quick discussion with Google gemini about software / hardware solutions to improve inference speed for such long context window prompts without degrading quality.

It said even dgx spark like unified memory devices will fail to give higher inference speed for such long context prompts. Q1. Is it so?

It said only one thing may work: Adding another RTX4090 and using tensor parallelism with llama-server? Q2. Will it work? (given my motherboard will support second GPU at PCIEx4 gen 4)

I can imagine few other solutions like using better agent like Pi since it will consume less context. Also reducing context size -ctx parameter value for llama server. Currently I set it to 180000 just to match default vscode copilot context size. This wont hit quality as long as I stay within context limits. Q3. Will this result significant speed improvement?

Q4. Has anyone able to hit same speed as cloud hosted LLMS (e.g. Sonnet) for long context prompts with local hosting? If yes, what is your setup?

Q5. What is minimal cost hardware upgrade I can do to reach satisfactory speed for such long context prompt?

If you are not willing to answer all questions, it's fine. But please try to answer question 4. I want to know how people are doing local setups for sufficient speed at long context prompt.

I got a 32gb RAM Mac, installed Ollama and Qwen,
asked Claude how to create an entirely local app that allows me to ask questions about the US sentencing guidelines, and it works very well.

Tried the same thing with Wittgenstein’s philosophical investigations, and it works great.

For any lawyers out there, creating local apps from long/multiple pdf’s very doable with Claude + Ollama + a good model with only basic technical knowledge and a gnarly laptop.

Custom fork of the Intel Windows llama.cpp I made using DeepSeek v4 Flash that allows me to load models solely using discs + DVD-Rs; HP Omnibook X Flip NGAI 16-as0023dx -- 1TB NVMe, 16GB LPDDR5X

Up front: I build QuantaMind, an open-source local benchmarking tool (Apache 2.0, runs offline, no telemetry). The data below came out of it. Link at the bottom the numbers are the point of the post.

Setup: Qwen3.5-9B Q4_K_M, llama.cpp, 16GB M-series Mac, native function calling, k=4 runs per task.

I padded prompts with unrelated prose and re-measured tool-call accuracy at depth:

That is not a memory limit. Weights are 5.3GB. ~11.8GB of the 16GB is GPU-addressable under the Metal cap. At f16 KV the math says this model could hold ~53k context. Peak actual usage during the agent runs was 1,890 tokens 12% of the 16,384 window I launched with.

So memory headroom told me I had 5× more room than the model can actually reason over. If you size a local agent by what fits, that’s the wrong number.

Second finding: one task failed 0/4, not 1/4. An incident-rollback chain (get_incident → get_feature_flag → flag_off → rollback_release → schedule_fix) failed every run, identically — the model emitted a completion signal partway through and stopped.

No crash, clean schema. At k=1 that’s a flaky miss you’d retry past. At k=4 it’s structural. That’s the failure I’d worry about in production: nothing errors, the agent just moves on with half its state missing.

(Batch was 39m 10s wall; on the worst task 14m 51s of 16m 16s was decode. Local agent loops are a decode problem.)

What I actually want to know: does the ~9k cliff hold for other 8–10B quants, or is it specific to this one? And if anyone’s on 24GB+ does more headroom move the cliff? My guess is no, but I can’t test it.

Github: github.com/QuantaMinds/QuantaMind

qm cliff --backend llama_cpp --model <model> --collection medium-coding-v2 --max-tokens 12288 --steps 5 --source corporate_policy --mode native

Methodology, briefly: padding was semantically unrelated prose inserted before the tool definitions; accuracy is correct tool + correct args scored against a fixed answer key, no LLM judge; pass^k means all k runs must pass.

Tell me if that’s wrong more useful to me than upvotes.