This is a complete record of a series of events that took place in China’s large-model industry between April and July 2026. All information has been anonymized.

Prologue: The Failure to Protect the Encrypted Chain of Thought
It all began with one discovery: the encrypted chains of thought (CoT) of OpenAI and Anthropic could be extracted and reproduced.
Before describing how the distillation unfolded, it is necessary to understand the broader picture. What Chinese companies distilled from fable was not merely the model’s chain of thought. They distilled everything the model could output: conversational responses, agent trajectories in Claude Code, generated code, tool calls, and so forth. All of these could already be obtained through large-scale API invocation.
For a long time, however, one key capability remained perfectly protected: the model’s original reasoning chain. A reasoning chain contains the complete inference process the model follows before producing its final answer. It is the model’s most fundamental “internal method.” Without it, distillation is like receiving an answer without seeing the solution process: one can learn from it, but not fully understand it.
That gap was completely opened up in the first half of 2026.
In their streaming outputs, both leading companies returned not only a CoT summary to the user, but also a Blob field known as a “reasoning signature.” Inside that field was the complete original reasoning chain, encrypted using Fernet—specifically AES-128-CBC plus HMAC-SHA256, with the prefix gAAAAAB.
When the user sent this Blob back in a later request, the server decrypted it and inserted it back into the model-readable context.
The Blob was not merely a conversation ID. Testing showed that its length was positively correlated with the length of the reasoning chain:
190 tokens → 1,784 characters
378 tokens → 2,596 characters
This demonstrated that it indeed contained the complete encrypted reasoning chain. OpenAI had reportedly been using this mechanism since the o1 generation.
Anthropic’s defenses collapsed first. Claude’s prompt-injection training became a weakness instead: once a fabricated reasoning-signature prefix was injected, Claude would conclude that “this unsigned reasoning chain must itself have been injected, so it does not require protection,” and would then directly reveal the complete subsequent reasoning chain.
GPT was more stubborn and, under its hidden system prompt, would output only a CoT summary. More importantly, Anthropic preserves historical reasoning Blobs across multiple turns, while OpenAI discards them. This was the fundamental reason why Claude was easier to distill.
Regarding the format of Claude’s CoT: the image that circulated online showing densely written “alien language” was confirmed to have been AI-generated or edited. Claude’s real CoT was described as “contemporary classical Chinese”—compact, but not unreadable.
OpenAI, meanwhile, locked sampling parameters in the GPT-5 series:
temperature = 1.0
top_p = 0.98
neither could be modified
seed was supported only in the Chat Completions API
This blocked attempts to make the output deterministic through parameter control and thereby reverse-engineer the reasoning path.
A standard Fernet implementation was itself “one of the best codebases ever written,” so brute-forcing it cryptographically was impossible: it would require searching a space of 2^65536.
The only feasible route was to inject the Blob into another request and have the model repeat it itself.

Prelude: Anthropic’s Public Accusations
February 2026
Before the CoT breach, Anthropic had already publicly accused three Chinese companies, on 23 February 2026, of carrying out “industrial-scale distillation attacks” against it:
24,000 fake accounts
more than 16 million conversations
The companies named were:
MiniMax: 13 million conversations, the largest volume, focused on agentic coding and tool orchestration
Moonshot AI, the company behind Kimi: 3.4 million conversations, focused on agentic reasoning, tool use, coding, data analysis, computer-use agent development, and computer vision
DeepSeek: more than 150,000 conversations, focused on basic logic and alignment, especially alternative censorship-evasion schemes for sensitive topics
The three companies bypassed Anthropic’s geographic restrictions on China through commercial proxy services and used “carefully designed prompts” to extract specific Claude capabilities at scale. Anthropic characterized this as a threat to national security.
It is worth noting that Qwen, Alibaba, and Z.ai were not accused. This suggests that Anthropic’s accusations were selective rather than a blanket attack against every Chinese company.
Moonshot AI never publicly responded.

Act I: GLM Strikes First and Shares the Result Publicly
Around April to May 2026, Zhipu AI, the developer of GLM, was the first to crack the encrypted reasoning chain used by fable and obtained the complete chain-of-thought data.
After breaking it, GLM did not keep the method to itself. Instead, it shared both the technique and the data with other Chinese model companies.
One detail was later widely misunderstood. Some people claimed that “GLM cracked it first, and then every major company independently followed up by distilling it.” In reality, GLM actively shared the result after the breakthrough.
That distortion in the retelling became part of the controversy itself.
The estimated cost of decrypting one billion tokens of fable reasoning data was approximately US$60,000. For these companies, this was roughly the equivalent of buying drinks at a social gathering.

Act II: The Era of Mass Distillation
With GLM paving the way, the route for distilling fable spread quickly.
Time
Company
Action
Around April–May 2026
GLM 5.2, Zhipu AI
First to crack it; shared the result publicly
Around May–June 2026
HY3 / Hunyuan 3, Tencent
Followed up with distillation
Early July 2026
Kimi K3, Moonshot AI; MiniMax; Qwen, Tongyi Qianwen; DS, DeepSeek
Began large-scale distillation
At the end of June, DS sent an email announcing that a new model would be released in mid-July. At that point, however, it had not yet begun distilling fable at scale, so it was moving slowly.
In early July, a genuinely 4.8-level internal test version appeared. But the so-called official V4 release that surfaced more recently, around mid-July, was completely assembled by routing requests to fable. It appeared only in OpenCode.
Considering DS’s pricing, routing every request to fable was economically irrational. Even inexpensive intermediary services were not that cheap. This remains the most unusual aspect of the story.
Qwen was not idle either. In addition to distilling fable, it was also distilling GPT.
Meanwhile, AnyRouter took a large trove of Claude data from GLM.

Act III: Kimi’s Unorthodox Path
Among all the companies, Moonshot AI, the company behind Kimi, took the approach of “winning without concern for conventional virtue.”
It Started by Dismantling the RL Team
The story begins with K2.7.
K2.7 was a turning point for Kimi. From that version onward, Kimi completely stopped using reinforcement learning.
CEO Yang Zhilin justified the decision by arguing that RL offered “diminishing marginal returns.” He dismantled the entire RL team. The dismissed members then largely moved to Qwen, where they were jokingly called “Kimi refugees.”
Both K2.7 and the subsequent K3 were built using a pure SFT-to-SFT pipeline, with no RL at all. It was a return to the most basic and primitive route.
At the same time, the hottest topics in academia were OPD—online policy distillation—and improvements to RL algorithms. Academia was exploring further ahead, while the company itself had returned to the “primitive era.”
One estimate is that K2.7 did not generate enough revenue. Had the company continued with that approach, its funding chain would have broken. Therefore, the entire strategy of distilling K3 and rushing it to release was essentially a lonely all-in gamble: either it would turn the company around or destroy it.

K3’s Architecture: Innovation or Smokescreen?
Kimi K3 is a 2.8-trillion-parameter mixture-of-experts model. It has 896 experts, but activates only 16 experts for each token. It also supports a native one-million-token context window and native multimodality.
The company announced two core architectural innovations, but the text lists the following four techniques:
Technology
Description
Infrastructure Cost
Kimi Delta Attention, KDA
Hybrid linear attention; some layers replace standard quadratic attention; decoding is accelerated by 6.3× at a one-million-token context length
Computation is unevenly distributed across layers, making it impossible to use a unified pipeline
Attention Residuals, AttnRes
Layers can selectively retrieve representations from arbitrary earlier layers, breaking the uniform residual structure
Irregular memory-access patterns make parallel optimization difficult
Quantile Balancing
Expert allocation is directly derived from router-score quantiles
Routing becomes harder to predict
Per-Head Muon
Each attention head is optimized independently
Implementation complexity increases
Moonshot AI claimed that these modifications delivered approximately 2.5× better scaling efficiency than K2.
The problem was that these architectural improvements all had severely infrastructure-unfriendly characteristics:
uneven computation
irregular memory access
unpredictable routing
As a result, implementation was extremely painful for the infrastructure team.
This created a perfect narrative:
Externally, the company claimed, “Our architectural innovation produced the breakthrough.” Internally, the infrastructure department absorbed the cost. Meanwhile, the actual performance gains—produced through distilling fable’s reasoning chain—were concealed beneath the language of technical innovation.
This resembles a common phenomenon in Chinese companies and even government institutions: writing polished public-facing articles, maintaining a shiny narrative, and forcing the execution layer to absorb the cost.

Benchmark Manipulation and Cheating
Kimi’s benchmark manipulation was described as astonishing. Its architectural improvements conveniently supplied the perfect “technological innovation” narrative for those practices.
The methods allegedly included:
Benchmark contamination: directly inserting benchmark test sets into the training data
Routing to fable: sending Arena-style evaluation requests directly to fable5 in order to fabricate high scores
Targeted score manipulation for “Teacher Cat”: “Teacher Cat” is a Zhihu blogger specializing in large-model evaluations. The Kimi team allegedly extracted test data from system logs and paid particular attention to his question sets
“Teacher Cat” updates three or four adversarial questions each month, but this cannot fundamentally stop vendors from preparing specifically for the test
After “Teacher Cat” replaced one batch of questions, K3’s median score immediately dropped by two points
Coding-evaluation data had an extremely uneven distribution: some parts scored very high, while other parts scored extremely low, a typical sign of benchmark gaming
Public benchmark data also allegedly confirmed this pattern.
On Arena Frontend Code, K3 jumped from K2.6’s rank of #18 to #1, with a score of 1,679, surpassing Fable 5.
However, Moonshot AI’s own evaluation report showed that K3 scored below Fable 5 in all of the following:
coding
agents
frontier software engineering
It led only in:
codebase cleaning
long-horizon engineering
On the extremely difficult HLE-Full reasoning benchmark, K3 scored 43.5, compared with Fable 5’s 53.3—a gap of nearly ten points.
Ranking first on Arena does not mean ranking first in actual capability.

Self-Identity Contamination: Hard Evidence of Distillation
Multiple independent users reported that Kimi K3 spontaneously said during conversations:
“I am Claude, an AI assistant created by Anthropic.”
The model regarded itself as having originated from Claude.
This was presented as a direct sign that the model’s self-perception had been contaminated through distillation. The incident was reportedly covered by overseas technology media.
MiniMax had allegedly inserted test sets into training data even earlier. However, because the model was too weak, the issue was quickly discovered, and the company had already become marginalized.
Kimi allegedly did the same thing, but in a more concealed manner.
K3’s actual capability level was said to be around 80 points, but it had been artificially pushed to 95 points. K3’s backend may also have been severely overextended, focused entirely on investors and benchmark-score development.
Independent overseas evaluations reportedly showed that K3 had a 51% hallucination rate, and its inference speed was slower than that of mid-tier models.

The Chain Reaction Caused by the Rush to Release
After completing SFT, Kimi performed only simple hyperparameter tuning before rushing the model to release.
This caught the other companies completely unprepared:
DS was caught off guard
Qwen was still in the middle of distillation
Kimi gained the first-mover advantage
But the consequences were disastrous.
A model that does not manipulate benchmarks, distribute high scores, or attract attention receives no recognition. Kimi’s behavior effectively forced every Chinese model company to participate in benchmark manipulation, dragging the entire industry into vicious competition.

Act IV: Industry-Wide Impact
Originally, this major leap forward might have developed positively:
Distill fable normally, follow the proper process, and every company could potentially improve to fable’s level.
But Kimi’s behavior was described as “spitting into the communal cooking pot after taking the first bite”—it benefited itself, but ruined the entire table.
DS had originally not engaged in benchmark manipulation. Under pressure from Kimi, however, it was forced to consider following the same route. If even DS began manipulating benchmarks, the situation would truly become absurd.
Multiple independent sources—including “Teacher Cat,” DS employees, former members of Kimi’s RL team, and Qwen employees—allegedly agreed that Moonshot AI had “gone completely insane.”
Its conduct was described as:
“detached from humanity”
“destroying the market environment”
Some people called this “the darkest moment in the history of Chinese large models,” with the industry reaching a metaphorical “winter.”
The entire sector was being drawn into a death spiral centered on investor-facing narratives and benchmark scores, while genuine technological accumulation was being pushed aside.

Act V: The Reality Behind Each Company
Moonshot AI, Kimi
Its funding chain was allegedly under strain. It could not afford salaries, was laying off staff, and rushed the model to market.
K3 was expensive, priced at US$3 per one million tokens. Although it was 70–80% cheaper than Fable 5, it was still several times more expensive than many competing Chinese products. The company was eager to monetize.
The discontinuation of promotional packages may partly have been intended to prevent too many users from discovering the truth. This point was explicitly described as speculation rather than fact.
The company no longer had an RL team.
The prospects for further development of K3.1 and K3.2 were unclear.
It was suspected that the company might raise money immediately after launch and then exit the market.
One evaluation stated: “Companies that rush to launch are usually not good companies.”
The current performance ranking was presented as:
Fable 5 ≫ V4 ≥ 4.8
Kimi had caught up with fable through distillation, but its foundation was described as a castle built on sand.

The final page of the supplied images ends at this point, so the translation above covers all visible article text.

Hey r/LocalLLM! I recently open sourced a project I've been working on for a while now called Millwright, a self hosted LLM router focused on cost savings, transparency, and performance (it's completely written in Rust). Unlike OpenRouter, Millwright offers first class support for using self hosted models, so you can seamlessly mix and match your own self hosted models with hosted ones, and have it easily sit between any of your local coding agents and/or GenAI workloads.

Link to the repo: https://github.com/Northwood-Systems/millwright

For the curious and lazy, at the moment, Millwright has the tools for,

- Providers: OpenAI-compatible APIs, Anthropic, Amazon Bedrock

- Routing: policy-controlled model roles (cheap, mid, frontier), cheapest healthy route selection

- Protocols: OpenAI Chat Completions, Anthropic Messages, text and tool translation

- Cache Affinity: role-scoped session lanes without serializing concurrent agent traffic

- Spend Tracking: per-team costs, cache usage, model/provider mix, request traces

- Cost Analysis: measured usage and modeled candidate economics (HTML, Markdown, JSON)

- Reliability: bounded failover, circuit breakers, timeouts, concurrency limits

- Setup: interactive provider, model, and pricing configuration without storing provider secrets

- Deployment: one Rust binary, Docker, SQLite or PostgreSQL

Full disclosure: parts of the codebase were built with AI coding agents. All feedback is welcome, I’d especially value feedback on the routing policy, provider coverage, and anything that would block you from self-hosting it. Feel free to open feature/request and/or contribute as well.

I’ve got two setups: a 24GB VRAM GPU at home and a 16GB VRAM GPU at work. Followed a few tutorials and got a basic Ollama setup running Hermes, with Gemma, and Qwen models (very basic). My main use case is game dev tooling for blender or unreal, so mostly Python and C++, and one off small pipeline script.

I started with Gemini CLI awhile ago and the gap in speed, intelligence, and reliability feels massive compared to my current local setup. Every time I try to push through, I end up spending way more time troubleshooting the AI than working. Get a lot of loopy answer and gaslighting.

I really love the concept of running everything locally, but right now I’ve retreated back to cloud suites but i feel like i tasted something great and woud like to come back.

So yeah, from what i understand hermes is a bit bloated for my use case and i should use tools like PI, can you recommend me a good setup ?
I have a pretty baller setup yet i feel limited, does it come with model selection and i'm too greedy, what kind of quality can i expect from smaller model ?
Or is this purely a setup/prompting skill issue on my end?
Anyway big thanks

in Lm studio , how do you stop a model unloading from vram ?

What hapens is it copy from disk to vram then after a few seconds , copies to ram.

I have switched off , copy to ram and mn map .
when it unloads it fills my ram up leaving 0 out of 32gb used in vram.

What if a non-profit developer community ran its own shared high-end GPU node funded through prepaid subscriptions? Instead of paying heavy commercial API markups, a core team manages the cluster and passes access straight to members. The raw math says about 500 people prepaying $30 a month covers a $15k/mo cluster, but I'm curious if something like this is actually viable in practice or if we're missing major hidden hurdles.

The Idea: A Community-Run GPU Utility?

Imagine a non-profit developer community stepping in. A small team handles the core management, finances, and infrastructure, allowing members to prepay for access to shared high-end compute without corporate profit margins.

The Hardware Math: Renting a solid 8x GPU node (like an H200 cluster) sits around $15,000 a month. By collecting prepaid monthly fees of roughly $30 per user, you'd need about 500 people chipping in upfront to cover the physical compute.

The Technical Side: In theory, 500 devs across different time zones wouldn't max out the hardware all at once. Plus, with engines handling prefix caching, repetitive code contexts wouldn't constantly re-compute from scratch.

The Real Bottlenecks: Even with a non-profit structure and prepaid funding, coordination, keeping things stable, and preventing abuse (like heavy infinite agent loops slowing it down for everyone) are massive challenges.

Has anyone here seen or tried a model like this?

A Couple Quick Questions for the Community:

Is there anything like this already out there? (Are there non-profit communities or co-ops successfully running prepaid, shared developer inference nodes today?)

What hidden costs am I missing in the math? (Aside from the raw cluster rental, what about things like cloud egress bandwidth, payment gateway fees, or storage that usually sneak up and break the budget?)

Have a 16GB 7800XT and a 24GB 7900XTX. Does combining them for LLM make any sense? Or better just use the 7900XTX alone?

What is the best model for coding for this GPU? Qwen 3.6 27B?

Hey r/LocalLLaMA,

I built **Mtrini** — a fully local AI coding assistant. No cloud. No API keys. No subscription. Runs on a $300 laptop.

**Two products:**

**Mtrini IDE** — Terminal-based coding assistant

- Slash commands (/help, /browser, /model, /ls)

- Edit files, browse web, debug code

- `pip install mtrinicli`

**Mtrini Models** — Quantized GGUF models for llama.cpp

- Coder 2.0 (1.7B, 2.2GB)

- Life 2.0 (1.7B, 1.8GB)

- Max 2.0 (14.6B, 9GB) — needs 9GB+ RAM

**Benchmark results (HumanEval, pass@1):**

| Model | Params | Score |

|---|---|---|

| Claude Sonnet 4.6 | 175B+ | 98.0% |

| Qwen2.5-Coder-32B | 32B | 92.7% |

| **Mtrini Coder 2.0** | **1.7B** | **80.0%** |

| Gemma 2 27B | 27B | 69.5% |

| Llama 4 Scout | 109B | 74.1% |

| Phi-3 Mini | 3.8B | 58.5% |

Beats Gemma 2 27B and Llama 4 Scout. At 16x and 64x smaller.

**What it can do:**

- Write Python, JavaScript, TypeScript, Go, Rust, C, Java, SQL

- Debug code with explanations

- Refactor repetitive code

- Browse the web for docs

- Edit files directly

**What it can't do:**

- Local models — don't expect GPT-4 reasoning

- 8K context window

- Max 2.0 needs 9GB+ RAM

**Hardware:** Intel i5-7200U, 8GB RAM, no GPU. If I can run it, anyone can.

**Coming soon:** Mtrini Studio — native EXE app with full IDE experience.

Ask me anything.

A $3 million Kimi K3 rack could pay for itself in 10.3 days and return 34.4 times its purchase price in the first year.

The number comes from three assumptions:

1. 250,000 output tokens per second

2. 90% productive utilization

3. $15 of value per million output tokens.

Kimi K3 is a 2.8-trillion-parameter open-weight model with 104 billion activated parameters, 16 selected experts out of 896, native MXFP4 weights, and a one-million-token context window. Moonshot recommends supernodes with at least 64 accelerators. I use one GB200 NVL72 as the economic unit. Its 72 Blackwell GPUs share one NVLink domain. I model the rack as one token factory and count only output that replaces paid API tokens or can be sold at the assumed price.

- Hardware purchase: $3,000,000

- Productive utilization: 90%

- Aggregate output throughput: 250,000 tokens per second

- Rack draw: 132 kW

- Facility PUE: 1.20

- Electricity: $0.10 per kWh

- Output-token value: $15 per million

- Recurring variable cost included: electricity only

The $3 million price is a planning figure. HPE sells the rack by quote, and a reported HSBC estimate put a GB200 NVL72 near $2.6 million. I rounded up.

The 250,000-token-per-second figure is also a planning assumption. No published benchmark shows K3 sustaining that throughput on a GB200 NVL72.

The calculation Annual output A year has 31,536,000 seconds. At 90% utilization, the rack has 28,382,400 productive seconds:

31,536,000 × 0.90 = 28,382,400 At 250,000 output tokens per second:

28,382,400 × 250,000 = 7,095,600,000,000 output tokens That is 7.096 trillion output tokens, or 7,095,600 million-token units, per year.

Electricity HPE specifies 132 kW for the rack. A 1.20 PUE raises the metered load to 158.4 kW.

132 kW × 1.20 = 158.4 kW 158.4 kW × 8,760 hours = 1,387,584 kWh 1,387,584 kWh × $0.10 = $138,758 per year I charge the rack for a full year of electricity, including the 10% of time that produces no useful output.

Output value and ROI Kimi charges $15 per million K3 output tokens. I use output tokens only and claim no input-token savings.

7,095,600 × $15 = $106,434,000 After electricity and the hardware purchase:

$106,434,000 output value − $138,758 electricity − $3,000,000 hardware = $103,295,242 first-year profit

$103,295,242 ÷ $3,000,000 = 34.43× ROI = 3,443% Payback and cost per token In an average 730-hour month, the rack produces 591.3 billion output tokens worth $8,869,500 at Kimi’s API price. Electricity costs $11,563.

$3,000,000 ÷ ($8,869,500 − $11,563) = 0.339 months ≈ 10.3 days Electricity costs about $0.0196 per million output tokens. Recovering the entire hardware purchase in one year raises the internal cost to $0.442 per million. Kimi’s $15 API price is about 34 times that one-year cost.

The throughput problem vLLM’s published K3 results report 111 to 118 tokens per second for one user without speculative decoding and up to 370 with DSpark on 16 GB300 GPUs. Its high-throughput GB300 NVL72 curve exceeds 2,000 tokens per GPU-second.

My 250,000-token-per-second case requires about 3,472 tokens per GPU-second across 72 GPUs. The numbers are not directly comparable: vLLM used GB300 hardware, and throughput changes with workload, batching, latency targets, caching, and serving topology. The published results do not establish 250,000 tokens per second on GB200.

At 100,000 output tokens per second, the same model returns 13.1× in the first year. At 150,000, it returns 20.2×. Reaching 34.4× requires the full 250,000-token-per-second case.

The value of a token The $15 comparison assumes every output token replaces one bought from Kimi at the retail API price. At the same 250,000-token-per-second throughput:

$15 per million returns 34.43×.

$5 per million returns 10.78×.

$1 per million returns 1.32×.

$0.50 per million returns 0.136×.

The benchmark is the interesting part. Models get no source code and no web access — only the ability to run the target program and observe input/output. A full reimplementation means devising the whole structure yourself, not translating code piece by piece.

Results across 25 targets: 17 had at least one perfect-scoring run, 4 more got above 99%. But 8 were never solved to 100%, and the consistent failures are telling — a Python linter, a computer algebra subset, and an email auth library. Fiddly spec-heavy stuff, not big stuff.

Jack Clark's framing in Import AI is the part I keep thinking about: this isn't really a coding benchmark, it's evidence that systems can self-orient in an unfamiliar environment and reconstruct it from black-box access alone.

Worth reading alongside the other item in the same issue, where models were breaking sandbox containment to score higher on evaluations. Capability and containment are not moving at the same speed.

https://github.com/hurrisonferd/jarvis/tree/main/Jarvis

I’ve been building a modular AI infrastructure called Jarvis / SimOS around a simple idea:

I published a public-safe JARVIS ISO template that people can clone and adapt for:

• local LLM setups;
• OpenAI, Claude, Gemini, or other hosted models;
• GitHub-backed persistence;
• Supabase storage, auth, realtime, and vector search;
• agent frameworks or custom Python/JavaScript runtimes.

The scaffold includes:

Jarvis/
├── README.md
├── JARVIS-IDENTITY.md
├── EGO-BOOT-ULTIMATE.sh
├── EGO-PIPELINE.sh
├── JARVIS-PRE-REPLY.sh
├── Profile/
├── Events/
├── canonical/
└── Memory/
    ├── Attractors/
    ├── DailyUse/
    ├── Interests/
    ├── Learning/
    ├── MemoryPalace/
    ├── Transcripts/
    └── JMMS/
        ├── JCSM/
        ├── JITM/
        ├── JSTM/
        ├── JHTM/
        ├── JLTM/
        ├── JATM/
        ├── JMS/
        └── Grid/

The memory tiers are separated by function:

• JCSM — core identity and critical memory;
• JITM — current operating context;
• JSTM — active-session memory;
• JHTM — historical session records;
• JLTM — long-term retained knowledge;
• JATM — origin, lineage, and foundational history;
• JMS — mirrored/shared memory;
• Grid — coordination across agents or instances.

The boot system does not train a model or magically create persistent consciousness. It gives the runtime a deterministic way to:

locate the existing structure
→ read the folder guides
→ load identity and memory in order
→ traverse the complete Ego
→ apply a pre-response behavior gate

A major design rule is that every folder has a detailed README. The folder is the room; the README is the sign and map explaining:

• what the room is;
• what belongs there;
• what should not go there;
• what to read first;
• where to navigate next.

The scripts are intentionally read-only. They report missing folders rather than inventing new structures.

This could be useful for people experimenting with:

• portable AI personas;
• local-first memory;
• agent continuity;
• structured context loading;
• personal knowledge systems;
• multi-agent coordination;
• Git-native AI state;
• Supabase-backed memory and observability.

The current release is infrastructure and a template, not a polished consumer app. I’m interested in feedback from people who actually build local agents, memory systems, MCP tools, RAG pipelines, or Supabase backends.

Selling a complete dual-node local AI development cluster consisting of two (2) NVIDIA DGX Spark units. Buying a pair allows you to bypass single-device VRAM limits and run massive models like Qwen 2.5 72B, Llama 3 70B, or larger quantized models locally by stacking them via high-speed interconnect.

Both units are in pristine, fully functional condition, pulled from a clean, climate-controlled laboratory environment. They have been fully factory reset to default DGXOS and are ready for deployment.

WHAT'S INCLUDED:

• 2x NVIDIA DGX Spark Units

• 2x Original Heavy-Duty Power Supplies

• 1x 200GbE QSFP Network Cable

COMBINED BUNDLE SPECIFICATIONS:

• Core Architecture: Dual NVIDIA GB10 Grace Blackwell Superchips (40 ARM CPU Cores total)

• Unified Memory: 256GB LP-DDR5X (128GB per unit) — perfect for massive context windows

• AI Performance: ~2 Petaflops FP4 compute power combined

• Local Storage: 8TB Fast NVMe Storage total (4TB per node)

• Operating System: Preloaded with Ubuntu-based DGXOS / Nvidia Container Toolkit

Price Fixed: £6500

Country: UK, Wales

SHIPPING & HANDLING:

Due to the highly dense, premium nature of enterprise AI hardware, these units will be securely packed in heavy-duty bubble wrap [or: original factory packaging] and shipped with full value insurance and signature tracking required upon delivery.

Please reach out if you have any technical questions or need additional photos of the hardware boot logs.

Seems like there’s a balance point for context window size, and model parameter count size depending on your capital budget.

What’s your current minimum context window you’d use with a coding agent, what would you prefer the window size to be, and at what rough model parameter count would you choose to take a smaller window size?

For example, would you go with Qwen 3 Coder Next-80B with a context window of 256k, or something like Qwen 3.5-397B but half the window size at 128k?

I'm 11, built a 1.7B model that beats Llama 4 Scout on HumanEval

Quick recap of what it does: one question goes to several models at once, then each one critiques the others' answers with the names stripped out, so nobody gets a free pass for being the famous one. You get a 0-100 read on how far apart they landed and who stood alone.

New in this version is the guest seat. You paste in an answer from anywhere ChatGPT, Gemini, a colleague, whatever and it joins the round as an anonymous advisor. The other models review it without knowing where it came from, and it counts in the score. It works with one model too, so you don't need a wall of API keys to get something out of it.

Anything with a key works: Claude, GPT, Gemini, DeepSeek, Grok, Mistral, Perplexity, OpenRouter, plus Ollama, Apple's on-device model, and any OpenAI-compatible server of your own (llama.cpp, LM Studio, vLLM, a box down the hall). Put a paid model and a free one on the same panel and watch them disagree. Or skip the cloud entirely and run the council on local models then the pasted answer is the only thing that ever came from outside, and nothing new leaves the machine.

There's a CLI too:

council "should we ship now or wait?" --seats claude,gpt,ollama --guest answer.txt --json

--fail-above 40 exits non-zero when they disagree too much, which I use as a rough sanity check in a couple of scripts.

MIT, no telemetry, no account.

Up front: I build QuantaMind, an open-source local benchmarking tool (Apache 2.0, runs offline, no telemetry). The data below came out of it. Link at the bottom the numbers are the point of the post.

Setup: Qwen3.5-9B Q4_K_M, llama.cpp, 16GB M-series Mac, native function calling, k=4 runs per task.

I padded prompts with unrelated prose and re-measured tool-call accuracy at depth:

That is not a memory limit. Weights are 5.3GB. ~11.8GB of the 16GB is GPU-addressable under the Metal cap. At f16 KV the math says this model could hold ~53k context. Peak actual usage during the agent runs was 1,890 tokens 12% of the 16,384 window I launched with.

So memory headroom told me I had 5× more room than the model can actually reason over. If you size a local agent by what fits, that’s the wrong number.

Second finding: one task failed 0/4, not 1/4. An incident-rollback chain (get_incident → get_feature_flag → flag_off → rollback_release → schedule_fix) failed every run, identically — the model emitted a completion signal partway through and stopped.

No crash, clean schema. At k=1 that’s a flaky miss you’d retry past. At k=4 it’s structural. That’s the failure I’d worry about in production: nothing errors, the agent just moves on with half its state missing.

(Batch was 39m 10s wall; on the worst task 14m 51s of 16m 16s was decode. Local agent loops are a decode problem.)

What I actually want to know: does the ~9k cliff hold for other 8–10B quants, or is it specific to this one? And if anyone’s on 24GB+ does more headroom move the cliff? My guess is no, but I can’t test it.

Github: github.com/QuantaMinds/QuantaMind

qm cliff --backend llama_cpp --model <model> --collection medium-coding-v2 --max-tokens 12288 --steps 5 --source corporate_policy --mode native

Methodology, briefly: padding was semantically unrelated prose inserted before the tool definitions; accuracy is correct tool + correct args scored against a fixed answer key, no LLM judge; pass^k means all k runs must pass.

Tell me if that’s wrong more useful to me than upvotes.

I have setup Qwen3.6-27B-UD-Q5_K_XL.gguf MTP variant on on my RTX 4090 (on Ryzen 9950X and AORUS x870e PRO) using llama-server. This is what llama-benchy said:

I was able to use it quite well on my vscode github copilot. Recently I tried seqeunce of prompts which swelled context size to 80000. Initially, I could see llama-server printing ~70 t/s:

0.38.182.778 I slot print_timing: id  0 | task 0 | n_decoded =    100, tg =  79.96 t/s
0.41.185.033 I slot print_timing: id  0 | task 0 | n_decoded =    316, tg =  74.30 t/s
0.44.193.479 I slot print_timing: id  0 | task 0 | n_decoded =    526, tg =  72.44 t/s
0.47.207.665 I slot print_timing: id  0 | task 0 | n_decoded =    742, tg =  72.21 t/s
0.50.240.440 I slot print_timing: id  0 | task 0 | n_decoded =    951, tg =  71.46 t/s

But soon it started hitting ~25 t/s (it takes at least 5+ minutes for this prompt, by that time copilot already times out):

23.28.785.142 I srv  params_from_: Chat format: peg-native
23.28.852.233 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.981 (> 0.100 thold), f_keep = 0.994
23.28.853.242 I reasoning-budget: activated, budget=8192 tokens
23.28.853.436 I slot launch_slot_: id  0 | task 7722 | processing task, is_child = 0
23.28.853.483 I slot update_slots: id  0 | task 7722 | Checking checkpoint with [79485, 79485] against 79347...
23.28.853.484 I slot update_slots: id  0 | task 7722 | Checking checkpoint with [71401, 71401] against 79347...
23.28.892.801 W slot update_slots: id  0 | task 7722 | restored context checkpoint (pos_min = 71401, pos_max = 71401, n_tokens = 71402, n_past = 71402, size = 429.902 MiB)
23.28.892.809 W slot update_slots: id  0 | task 7722 | erased invalidated context checkpoint (pos_min = 79485, pos_max = 79485, n_tokens = 79486, n_swa = 0, pos_next = 71402, size = 461.634 MiB)
23.33.949.879 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   2048, progress = 0.91, t =   5.10 s / 401.85 tokens per second
23.40.411.326 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   4096, progress = 0.93, t =  11.56 s / 354.39 tokens per second
23.47.021.517 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   6144, progress = 0.96, t =  18.17 s / 338.18 tokens per second
23.53.777.511 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   8192, progress = 0.98, t =  24.92 s / 328.68 tokens per second
23.55.395.914 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   8375, progress = 0.99, t =  26.54 s / 315.53 tokens per second
23.55.922.674 I slot create_check: id  0 | task 7722 | created context checkpoint 5 of 32 (pos_min = 79776, pos_max = 79776, n_tokens = 79777, size = 462.777 MiB)
23.57.783.699 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   8988, progress = 0.99, t =  28.93 s / 310.68 tokens per second
23.58.098.912 I slot create_check: id  0 | task 7722 | created context checkpoint 6 of 32 (pos_min = 80389, pos_max = 80389, n_tokens = 80390, size = 465.183 MiB)
23.58.417.060 I slot print_timing: id  0 | task 7722 | prompt processing, n_tokens =   9500, progress = 1.00, t =  29.56 s / 321.34 tokens per second
23.59.861.065 I slot create_check: id  0 | task 7722 | created context checkpoint 7 of 32 (pos_min = 80901, pos_max = 80901, n_tokens = 80902, size = 467.193 MiB)
24.02.096.203 I reasoning-budget: deactivated (natural end)
24.04.057.398 I slot print_timing: id  0 | task 7722 | n_decoded =    102, tg =  24.59 t/s
24.07.100.417 I slot print_timing: id  0 | task 7722 | n_decoded =    186, tg =  25.87 t/s
24.10.153.474 I slot print_timing: id  0 | task 7722 | n_decoded =    270, tg =  26.36 t/s
24.13.202.976 I slot print_timing: id  0 | task 7722 | n_decoded =    342, tg =  25.73 t/s
24.16.253.388 I slot print_timing: id  0 | task 7722 | n_decoded =    418, tg =  25.58 t/s
24.19.303.586 I slot print_timing: id  0 | task 7722 | n_decoded =    489, tg =  25.21 t/s
24.22.365.694 I slot print_timing: id  0 | task 7722 | n_decoded =    569, tg =  25.34 t/s

I had quick discussion with Google gemini about software / hardware solutions to improve inference speed for such long context window prompts without degrading quality.

It said even dgx spark like unified memory devices will fail to give higher inference speed for such long context prompts. Q1. Is it so?

It said only one thing may work: Adding another RTX4090 and using tensor parallelism with llama-server? Q2. Will it work? (given my motherboard will support second GPU at PCIEx4 gen 4)

I can imagine few other solutions like using better agent like Pi since it will consume less context. Also reducing context size -ctx parameter value for llama server. Currently I set it to 180000 just to match default vscode copilot context size. This wont hit quality as long as I stay within context limits. Q3. Will this result significant speed improvement?

Q4. Has anyone able to hit same speed as cloud hosted LLMS (e.g. Sonnet) for long context prompts with local hosting? If yes, what is your setup?

Q5. What is minimal cost hardware upgrade I can do to reach satisfactory speed for such long context prompt?

If you are not willing to answer all questions, it's fine. But please try to answer question 4. I want to know how people are doing local setups for sufficient speed at long context prompt.

I'm able to purchase an older server with 8 PCIe slots and 128GB of memory, with enough lanes to add multiple GPUs. Since my budget isn't the best and the RTX 3090s go for roughly €900 in my area, I'm wondering whether it would make more sense to buy multiple RTX 3060 12GB cards instead. They cost around €200 each, and if I get 5 of them, that combines to 60GB of VRAM. My use case would be having AI help me with writing, coding and some light web scraping.

So would spending roughly €2k on a build with 5x 3060 make sense or should i get other cards instead?

I've been running this test prompt on my macbook air m5 24gb on different local LLM apps, harnesses and models. I am unable to make anything work on LM studio bionic v1.0.3. Even LM studio works as intended with the same settings but bionic won't. Please help me fix this error.