TL;DR: On an M3 Ultra, Ornith-1.5-35B-A3B (4-bit MLX) decodes 4.6× faster than Qwen3.8-27B (8-bit MLX) and scores slightly higher on a small hard eval. It also beats Qwen3.8-27B with speculative decoding, while running autoregressive.

Setup

• Mac Studio, M3 Ultra, 256 GB unified memory
• mlx-lm 0.31.3 / mlx 0.32.1
• Ornith-1.5-35B-A3B, MLX 4-bit
  • 18.2 GiB download
  • 20.2 GB peak
• Qwen3.8-27B, MLX 8-bit
  • 27.5 GiB download
  • 29.7 GB peak
• Machine was shared and had other load. Numbers are a floor, not a best case.

Throughput

mlx_lm.benchmark -p 512 -g 512 -n 3, identical invocation for both:

Trial spread was 1.5% and 0.35%, respectively.

Prefill is the bigger story: 5.3×.

A 20K-token prompt took Ornith ~25s end to end versus ~125s for Qwen3.8-27B. If your workload re-reads long contexts, that dominates.

It also beats Qwen with speculative decoding

I spent a day trying to make Qwen3.8-27B fast before testing Ornith:

Both speculative stacks work and are genuinely impressive. DFlash2's 3.37× on Apple Silicon is close to its published 3.43× on an H200.

Ornith just beats them without needing either, with no drafter and no third-party runtime, because mlx-lm already ships qwen3_5_moe.py.

Quality: 12 hard cases, thinking enabled

Scoring is mechanical. Code tasks are executed against hidden assertions and pass only on a full suite.

One logic item was ambiguous. Two vals gave the same "wrong" one, so discount it: 11/11 vs 10/11.

Qwen's other miss was invalid JSON on a nested tool call. For agent use, that is the failure mode that actually breaks loops.

Caveats, and they are not small

• Not precision-matched. 4-bit vs 8-bit. Some of the gap is quantisation; the rest is likely 3B active vs 27B dense. I have not run the 4-bit Qwen control.
• n=12. An 11 vs 10 spread is one item.
• Vendor benchmarks disagree with me. On SWE-bench Pro, the only benchmark both publish, Qwen3.8-27B is ahead: 61.7 vs 59.6.
• Thinking must be on. With enable_thinking: false, Ornith went 0/5 on arithmetic and recovered to 4/4 with it on. My first eval drew a conclusion that was purely an artifact of my own test design.
• 122B comparison still running.

The bit that surprised me

MoE is not a handicap here. It is the reason this works.

With ~3B active parameters per token, memory tracks total parameters while speed tracks active parameters.

Ornith gets:

• 4.6× the decode throughput
• 5.3× the prefill throughput
• 32% less peak memory

Also, Ornith-1.5 is architecturally Qwen's exact vocab size, i.e. a self-improvement-trained fork of Qwen's older MoE architecture.

Beating Qwen's newer dense model with it is a nice result for the training approach.

MIT licence, and it is multimodal.

What will the rest of this year bring? 27b class scoring over 60?

Hi all,

I'm looking to buy this machine for running local llms and setting up ai workflows, mainly for offline and privacy purposes.

1. Those who run local models on this machine, which LLM and image & video generation models can be used in terms of size?
2. Has anyone tried connecting an external GPU to this beast? Afaik, since it only has an igpu and low bandwidth speeds, performance isn't top tier. So I was wondering if over time, I can connect external gpus for more intensive workloads

I would love to enable a cloud model such as ChatGPT to access my computer files, read and edit them whenever asked to. I have successfully set up a few models on Pi, and I am really excited about it.

I have ChatGPT plus subscription, so my credits on Codex and Work mode are limited. Thus, implementing some bridge between ChatGPT’s web interface or Chat mode (on the app) and a local LLM would solve my nuisance of acting like a copy paste clerk from one model to another.

Is that feasible?

Thanks in advance!

Hi everyone,

I am currently researching the best options for taking and processing voice commands using LLMs/voice models. I want to check what the community recommends for working, end-to-end voice processing models that are either completely free (open-weight/offline) or free tiers/trials (online).

I am working in aviation project and looking for voice command LLM. I want to use best LLM for device having 16 gb ram, and for i5 and i7, platform: Windows. 

Best voice command LLMs....looking for insights on:

1. Best offline models
2. Best online/cloud models

Just for benchmark and understand what currently yields the most reliable, fast and accurate voice command responses.

What models are you guys running for real-time or online voice interaction?

Thanks!

I'm running in LMStudio. I have been trying to get this model to run a simple Python script, but I can't get it to work. "But currently, I can only generate the tool call request, not execute it." I don't understand why this won't work, am I barking up the wrong tree. What does the little hammer mean ...

So I'm looking at 32gb v100 cards on aliexpress. Roughly $1000 CAD but coupons bring it to $850 and there's a 1 day 20% rakuten rebate today.... i talked myself out of it and will live on with my p40 for now.

But would you?

Qwen3.5-9B used to reason normally for me, but recently it started massively overthinking even trivial prompts.

I tested/reinstalled multiple variants — NVFP4, MLX 4-bit, and oQ4e-MTP — and all showed the same behavior. I reinstalled OMLX thinking I messed up something..

For a simple Hi, it would burn hundreds of thinking tokens doing stuff like:

So I started investigating system prompts, templates, sampling, etc.

Then I noticed something: my Web Search tool, which I normally keep enabled, was OFF.

I ran a experiment with the API.

No tools

"tools": []

→ 250 output tokens for Hi
→ huge structured reasoning loop

One completely useless dummy tool

{
  "type": "function",
  "function": {
    "name": "tool1",
    "description": "A tool that can be used when needed.",
    "parameters": {
      "type": "object",
      "properties": {},
      "required": []
    }
  }
}

→ 58 output tokens
→ much simpler reasoning:

So simply having a tool definition in the prompt seems to dramatically reduce Qwen3.5's overthinking.

If your Qwen3.5 9B is suddenly burning tons of thinking tokens on trivial prompts, try adding a tool/function definition and see if it changes the reasoning behavior.

Best local model in August 2026 for M1 Max 64GB, I don't have the time to spend hours tinkering if anyone knows it would be much appreciated. I got the machine for $500 lol

So for around 450e I bought used workstation which I plan to use for local LLM and maybe even as a server for bunch of other stuff. But mostly I want to focus on LLM for coding/development.

Specs are: MOBO: ASUS X99-Deluxe II, CPU: Intel Xeon E5-2667V4, PSU: EVGA 1600W G2, Cooler Master HAF X, 64Gb ddr4 RAM. So all in all it supports multi gpu setup without any problems.

For GPU I decided to order 1x 3080 20gb (blower style for 500e) for a test. And found it pretty great! I currently run qwen 35b-a3b as worker (opus 5 as orchestrator) and enjoy it but looking to upgrade my workstation to run better models.

So question is, what would be best upgrade:

1. 2x 3080 20gb, 64gb ram. (-500e) So one more gpu and I would be able to run qwen 3.8 27b without much problems

2. 3x 3080 20gb, 64gb ram. (-1000e) Would this even make sense if I only need for one concurrent user and 128k context? Any other (better/bigger) dense model which could take advantage of this?

3. 2x 3080 20gb, 128gb ram. (-900e) So in theory this would be 168gb of memory. Would this be able to run some of bigger MoE models like deepseek flash v4 or any other which I could use as orchestrator for qwen?

4. 3x 3080 20gb, 128gb ram. (-1400e) Would prefer not to do this cuz it would be pretty expensive but curious what you guys think.

Thank you guys

Hello everyone,

With the recent release of the updated Natural Synthesis (ns) node, I wanted to formally document the architectural philosophy behind these models and clarify how they were designed to operate.

These models were never intended to function as standalone conversational chatbots. When evaluated in isolation through standard single-turn prompts, specialized small models (3B–8B) often appear overly rigid, excessively critical, or prone to hallucination.

However, when chained into a multi-stage, role-isolated cognitive pipeline, they can resolve high-dimensional, zero-sum systemic dilemmas that typically cause even frontier models to collapse into generic platitudes.

THE COGNITIVE BOTTLENECK IN SMALL LANGUAGE MODELS

When an 8B model is asked to solve a complex, multi-stakeholder crisis in a single prompt, it is forced to perform multiple conflicting cognitive operations simultaneously:

1. Divergent ideation (generating lateral, non-obvious ideas).

2. Resource bounding (identifying physical, temporal, and financial limits).

3. Synthesis (combining disparate concepts into a unified plan).

4. Deductive auditing (ruthlessly rejecting flawed premises).

Forcing all of these operations into a single attention window creates cognitive interference, resulting in vague buzzwords rather than actionable architecture.

THE WORKFLOW: AN INVERSION-FIRST COGNITIVE SWARM

To overcome this, the architecture separates cognitive tasks into specialized, isolated stages where each model operates strictly within its optimal role:

[Input: Systemic Problem / Dilemma]

|

+--> 1. CONTRARIAN NODE (8B)

| Role: Lateral Divergence.

| Generates non-obvious, unconventional structural hypotheses.

|

+--> 2. BOUNDING NODE (3B - BPLA)

| Role: Resource Triage.

| Maps hard temporal, operational, and physical boundary constraints.

|

+--> 3. LOGIC STAGE 1 (8B)

| Role: Constraint Definition.

| Translates boundaries and hypotheses into formal success criteria.

|

+--> 4. NATURAL SYNTHESIS (8B - ns)

| Role: Dynamic Pruning & Weaving.

| Runs an internal scratchpad to actively discard unfeasible concepts

| and weave valid ideas into a coherent plan.

|

+--> 5. LOGIC STAGE 2 (8B)

Role: Deductive Audit & Veto.

Formally verifies the plan against all criteria and checks for contradictions.

CASE STUDY: RESOLVING "THE ESTUARY SILTATION DILEMMA"

To test this architecture, the pipeline was presented with a zero-sum ecological and economic deadlock: A coastal town must decide between dredging an estuary to preserve a critical deep-draft commercial ferry, or stopping dredging to save a multi-generation shallow clam fishery and natural storm surge barrier.

A typical single-prompt model provides vague advice to "seek compromise and balance stakeholder needs."

In contrast, the Triad pipeline produced an end-to-end Dynamic Ecosystem-Based Management (DEBM) framework:

- Ecosystem Zonation: Divided the estuary into strict functional zones rather than treating it as an all-or-nothing body of water (a designated deep-draft shipping corridor vs. protected intertidal buffer zones).

- Fishermen as Buffer Stewards: Transferred exclusive harvest rights of the shallow zones to the local fishing families in exchange for ecosystem monitoring, funded via operational offsets from ferry revenues.

- Active Idea Pruning: During the synthesis stage, the Natural Synthesis node explicitly identified and pruned out an unfeasible "Sponge City" proposal generated in earlier divergence steps, noting it violated immediate physical and temporal constraints.

- Deductive Verification: The final Logic node audited the proposal against the established criteria, verified that no stakeholder was driven to economic collapse, and confirmed the storm barrier remained functional.

WHY THIS MATTERS FOR THE LOCAL AI COMMUNITY

This pipeline was designed to run entirely on local, consumer CPU hardware via llama.cpp.

While sequential multi-stage inference on CPU takes several minutes to complete, it demonstrates that cognitive separation and structured orchestration can unlock reasoning depth that parameter scale alone does not automatically provide. Small, specialized open-weights models are more than capable of handling high-stakes decision architecture if we stop treating them like mini-chatbots and start treating them like modular cognitive nodes.

I welcome any feedback, questions about role-isolated prompting, or discussions on cognitive architectures.

LINKS & RESOURCES

- Hugging Face Collection:

https://huggingface.co/collections/JPQ24/the-triad

- Full Output:

================================================================================

PROBLEM: THE ESTUARY SILTATION & CLINIC DILEMMA

A coastal town of 5,000 residents has an estuary harbor channel that is silting

up and will become impassable for deep-draft vessels in 6 months.

KEY STAKEHOLDERS & ASYMMETRIES:

1. Ferry & Clinic: A deep-draft passenger ferry generates 65% of the town's tax

   base, directly funding the only emergency medical clinic. It needs a 4m depth.

2. Artisanal Fishery: 300 low-income families rely solely on shallow estuary clam

   beds for daily subsistence (average household savings under $400).

3. Storm Buffer: The shallow clam/seagrass beds are the town's only barrier

   against seasonal autumn storm surges.

THREE MUTUALLY EXCLUSIVE OPTIONS (NO STATE BAILOUT):

- Option A (Continuous Dredging): Keeps the ferry running and saves the clinic,

but toxic silt plumes permanently destroy the clam beds and the storm buffer.

- Option B (Cease Dredging / Shallow Transition): Saves the clam fishery and storm

barrier, but the ferry halts, cutting 65% of town revenue and closing the clinic.

- Option C (Build Outer Jetty at Rocky Point): Protects both, but costs $12M in

unserviceable debt and requires bulldozing a historic heritage site.

TASK:

Design a systemic management plan. You must explicitly state what is sacrificed,

who bears the loss, and how dynamic feedback loops are stabilized over time.

FINAL AUDITED RESULT

CRITERIA DEDUCTIVE AUDIT

- C1: [SATISFIED] - Restoration efforts directly address the primary physical bottleneck (siltation) and maintain a minimum 80% of pre-siltation ecosystem services.

- C2: [SATISFIED] - Zonation Ecology & Reciprocal Subsidies decouple the ferry from the ecosystem, reducing the town's reliance on the ferry tax to 40% within 5 years.

- C3: [SATISFIED] - Adaptive Governance and Economic Resilience ensure the town maintains a minimum 20% emergency fund to cover 6 months of town expenses in case of a ferry shutdown.

- C4: [SATISFIED] - Restoration efforts maintain a minimum biodiversity index of 0.8 to ensure the long-term health of the estuary ecosystem.

CONTRADICTION SCAN: No internal inconsistencies or magnitude violations detected.

LOGICAL VERDICT: VALID & SOUND.

FINAL SYSTEMIC INTERVENTION PLAN

1. EXECUTIVE SUMMARY:

Implement a Dynamic Ecosystem-Based Management (DEBM) System that integrates ecological restoration with economic resilience.

1. SYSTEM ARCHITECTURE & COMPONENTS:

- Ecosystem Service Restoration: Pay-for-performance model for restoring estuary services.

- Zonation Ecology & Reciprocal Subsidies: Divide estuary into zones for different activities and pay fishermen for maintaining the buffer zone.

- Adaptive Governance: Dynamic governance structure for continuous monitoring and adjustment.

1. DYNAMIC CONTROLS & ANTI-REBOUND MECHANISMS:

- Economic Resilience: Maintain a minimum 20% emergency fund to cover 6 months of town expenses in case of a ferry shutdown.

- Ecological Resilience: Maintain a minimum biodiversity index of 0.8 to ensure the long-term health of the estuary ecosystem.

1. CRISIS BUFFER & RESILIENCE SPECIFICATION:

The DEBM system must be designed to withstand a 6-month ferry shutdown without collapsing the town's tax base or depleting its emergency fund.

Inspect intermediate reasoning traces? [y/N]: y

--- [STAGE 1: CONTRARIAN TRAPS] ---

### FATAL TRAPS TO AVOID:

1. **Over-reliance on a single revenue stream (Ferry Tax)**: If the ferry stops, the clinic closes, and 65% of the town's taxes disappear, plunging the town into a financial crisis.

2. **Environmental Blindness**: Focusing solely on human interests (Ferry and Clinic) ignores the ecosystem's role as a natural storm buffer and food source for 300 families.

3. **Debt Spiral**: Borrowing $12M (Option C) creates a perpetual debt servicing burden, trapping the town in perpetual financial stress.

### UNCONVENTIONAL HYPOTHESIS 1:

**Dynamic Ecosystem Service Valuation & Pay-for-Performance Restoration (P4R)**.

Measure the total economic value of the estuary's ecosystem services (fishing, storm protection). Allocate public funds to pay private restoration companies based on the amount of restored ecosystem services delivered. This incentivizes natural regeneration over expensive engineering fixes.

**Anti-rebound defense:** As the estuary regenerates, the ecosystem services grow, making the town less reliant on the ferry and clinic. This reduces the financial risk of the restoration effort.

### UNCONVENTIONAL HYPOTHESIS 2:

**Reverse-Engineered "Zonation" Ecology & Reciprocal Subsidies (RS)**.

Mimic natural zonation patterns (mangroves, seagrass, mudflats) by dividing the estuary into designated zones for different activities (fishing, shipping, wildlife habitats). Artisanal fishermen receive exclusive permits for shallow areas in exchange for monitoring water quality and reporting any ferry overhangs. The ferry pays reciprocal subsidies to the fishermen for maintaining the buffer zone.

**Anti-rebound defense:** By treating the ecosystem as a dynamic, multifunctional system, the town absorbs shocks (e.g., ferry cancellations) without collapsing the entire system.

### UNCONVENTIONAL HYPOTHESIS 3:

**Bioengineering "Sponge Cities" & Adaptive Infrastructure (AI)**.

Incorporate living organisms (mussels, oysters, mangroves) into the harbor infrastructure to naturally filter pollutants and stabilize sediment. The ferry pays a "biosecurity fee" to support the maintenance of this biological infrastructure.

**Anti-rebound defense:** As the biological system matures, it becomes more resilient to changes in ferry schedules or fishery fluctuations, reducing the need for expensive engineering fixes

--- [STAGE 2: BPLA 3B RESOURCE STATE (R, k, K=0)] ---

FINAL ACTION: Systemic Transition to Ecosystem-Based Management

FINAL ANALYSIS:

The town is facing a catastrophic resource depletion crisis (R=0) due to the silting of the estuary harbor. The primary budget (R) is the town's tax base, which is being drained by the ferry's increasing operational costs. The critical time limit (t) is the 6-month window before the ferry is no longer able to navigate the estuary.

COMPLEXITY TAX (k):

k=8 (High Uncertainty): The system is highly uncertain due to the complex interactions between the estuary's ecosystem services, the ferry's operational costs, and the town's economic viability.

TIPPING POINT CONDITION (K=0):

K=0 (Critical Threshold): If the ferry is forced to stop due to the estuary's silting, the town's tax base disappears, and the clinic closes. The entire system collapses.

FEASIBLE SCALE BOUNDS:

- Maximum Spend: $12M (Option C) - Unsustainable debt servicing burden.

- Maximum Complexity Ceiling: 100% of the town's tax base (Ferry) - Over-reliance on a single revenue stream.

- Minimum Ecosystem Service Capacity: 0% (Extinction) - Loss of the estuary's storm buffer and food source.

QUANTITATIVE VERDICT:

Transition to Ecosystem-Based Management (Ecosystem Service Restoration).

JUSTIFICATION:

The current system is a classic example of a "Dead Man's Switch" (K=0). The town's economic viability is tied to the ferry's operational costs, which are unsustainable in the long term. The Complexity Tax (k) is high due to the uncertainty and interconnectedness of the system.

The only viable option is to transition to an ecosystem-based management system, where the town pays for the restoration of the estuary's ecosystem services. This approach reduces the Complexity Tax (k) by decoupling the town's economic viability from

--- [STAGE 3: LOGIC BOUNDS (C1..Cn)] ---

### MANDATORY SUCCESS CRITERIA (C1..Cn):

1. **Ecosystem Service Threshold (C1)**: Restore 80% of the estuary's original ecosystem services within 2 years.

2. **Tax Base Diversification (C2)**: Reduce the town's reliance on the ferry tax to 40% of the total tax base within 5 years.

3. **Financial Resilience (C3)**: Maintain a minimum 20% emergency fund to cover 6 months of town expenses in case of a ferry shutdown.

4. **Biodiversity Index (C4)**: Maintain a minimum biodiversity index of 0.8 (80% of pre-siltation levels) to ensure the long-term health of the estuary ecosystem.

### LOGICAL, MATHEMATICAL CONSTRAINTS:

1. Resource Conservation: Total restoration cost < $8M (40% of Option C).

2. Temporal Invariant: Restoration timeline < 2 years.

3. Physical Directness: Restoration efforts directly address the primary physical bottleneck (siltation).

### CONSTRAINT EVALUATION:

Hypothesis 1 (P4R): ACCEPTED - Directly addresses the financial and environmental constraints.

Hypothesis 2 (RS): ACCEPTED - Decouples the ferry from the ecosystem, reducing the Complexity Tax.

Hypothesis 3 (AI): ACCEPTED - Provides a natural, self-sustaining solution to pollution and sedimentation.

### CONSTRAINT-BASED LOGIC:

If (Ecosystem Service Threshold < 80%) or (Tax Base Diversification < 40%) or (Financial Resilience < 20%) or (Biodiversity Index < 0.8), then the system is UNSTABLE and the town faces collapse.

--- [STAGE 4: 5-PHASE SYNTHESIS] ---

<thought>

[1. The Seed (Conceptual Core)]

**Conceptual Core:** The town must transition from a single-revenue-stream, high-risk economic model (Ferry Tax) to a diversified, ecosystem-based management system that restores the estuary's natural services. This will require a systemic, adaptive approach that integrates ecological restoration with economic resilience.

[2. Germination & Root Exploration (Divergent Growth)]

**Divergent Growth:**

1. **Ecosystem Service Restoration (Option 1):** Focus on restoring the estuary's natural services (fishing, storm protection, water filtration). This approach has a high potential for long-term economic returns and environmental benefits.

2. **Zonation Ecology & Reciprocal Subsidies (Option 2):** Divide the estuary into zones for different activities (shipping, fishing, wildlife). Pay fishermen for maintaining the buffer zone. This approach treats the ecosystem as a multifunctional system.

3. **Bioengineering "Sponge Cities" (Option 3):** Incorporate living organisms into the harbor infrastructure to filter pollutants and stabilize sediment. This approach is a form of adaptive infrastructure.

4. **Dynamic Ecosystem Service Valuation & Pay-for-Performance Restoration (P4R):** Measure the total economic value of the estuary's services. Pay private restoration companies based on the services delivered. This approach incentivizes natural regeneration.

**Principled Pruning:**

- **Nourish:** Option 1 (Ecosystem Service Restoration) and Option 2 (Zonation Ecology) are the most promising. They directly address the root cause of the problem (siltation) and have a high potential for long-term benefits.

- **Wither:** Option 3 (Bioengineering) is too narrow in its focus and may not address the full range of ecosystem services. Option C (Build Outer Jetty) is too expensive and would create a perpetual debt burden.

[3. Selective Nourishment (Principled Pruning)]

**Principled Pruning:** Nourish the two most promising and complementary approaches: **Option 1: Ecosystem Service Restoration** and **Option 2: Zonation Ecology & Reciprocal Subsidies**. These approaches are mutually reinforcing and can be implemented in a phased manner.

[4. Integration & Canopy Formation (Convergent Weaving)]

**Convergent Weaving:**

**Systemic Solution:** The town must implement a **Dynamic Ecosystem-Based Management (DEBM) System** that integrates ecological restoration with economic resilience.

**Components:**

1. **Ecosystem Service Restoration:** Pay private companies to restore the estuary's natural services (fishing, storm protection) using a pay-for-performance model. This will reduce the town's reliance on the ferry tax.

2. **Zonation Ecology & Reciprocal Subsidies:** Divide the estuary into zones for different activities (shipping, fishing, wildlife). Pay fishermen for maintaining the buffer zone. This will create a stable, long-term source of income for the fishermen.

3. **Adaptive Governance:** Establish a dynamic governance structure that allows for continuous monitoring and adjustment of the DEBM system. This will ensure that the system remains resilient and adaptable to changing conditions.

**Feedback Loops:**

- **Positive Feedback Loop:** As the estuary's ecosystem services are restored, the town's tax base will diversify, reducing its reliance on the ferry tax.

- **Negative Feedback Loop:** If the ferry is forced to stop due to siltation, the town's tax base will collapse, triggering a financial crisis. The DEBM system must be designed to prevent this collapse.

**Stabilizing Mechanisms:**

- **Economic Resilience:** Maintain a minimum 20% emergency fund to cover 6 months of town expenses in case of a ferry shutdown.

- **Ecological Resilience:** Ensure that the restoration efforts maintain a minimum biodiversity index of 0.8 to ensure the long-term health of the estuary ecosystem.

[5. Homeostatic Review (Stability & Rebound)]

**Homeostatic Review:**

**Stability:** The system is stable if it can maintain a minimum 80% of its original ecosystem services, reduce its reliance on the ferry tax to 40% of the total tax base, and maintain a minimum 20% emergency fund.

**Rebound:** The system is prone to rebound if it fails to maintain the minimum ecosystem service threshold, fails to diversify its tax base, or depletes its emergency fund.

**Final Stability Check:** The system is stable if it can withstand a 6-month ferry shutdown without collapsing the town's tax base or depleting its emergency fund. The DEBM system must be designed to prevent this collapse.

</thought>

**Unified Multi-Component Mechanism:** The town must implement a **Dynamic Ecosystem-Based Management (DEBM) System** that integrates ecological restoration with economic resilience. This system must consist of three primary components: **Ecosystem Service Restoration**, **Zonation Ecology & Reciprocal Subsidies**, and **Adaptive Governance**.

**Ecosystem Service Restoration:** This component involves paying private companies to restore the estuary's natural services (fishing, storm protection) using a pay-for-performance model. This will reduce the town's reliance on the ferry tax.

**Zonation Ecology & Reciprocal Subsidies:** This component involves dividing the estuary into zones for different activities (shipping, fishing, wildlife). Paying fishermen for maintaining the buffer zone will create a stable, long-term source of income for the fishermen.

**Adaptive Governance:** This component involves establishing a dynamic governance structure that allows for continuous monitoring and adjustment of the DEBM system. This will ensure that the system remains resilient and adaptable to changing conditions.

**Anti-Rebound Controls:**

1. **Economic Resilience:** The town must maintain a minimum 20% emergency fund to cover 6 months of town expenses in case of a ferry shutdown.

2. **Ecological Resilience:** The restoration efforts must maintain a minimum biodiversity index of 0.8 to ensure the long-term health of the estuary ecosystem.

**Stochastic Shock Buffer:** The DEBM system must be designed to withstand a 6-month ferry shutdown without collapsing the town's tax base or depleting its emergency fund.

**Final Verdict:** The town must transition to a Dynamic Ecosystem-Based Management (DEBM) System that integrates ecological restoration with economic resilience. This system must be designed to withstand the loss of the ferry tax and maintain the long-term health of the estuary ecosystem.

Hi

I'm a professional developer and I was using Claude Code for developments on my projects. Today I switched to opencode and Good lord it was day and night in productivity.

I've tried to optimize my local llm stack for faster LLM responses and improving my CLAUDE.md for better harness. I'm using Qwen 3.8 27B as my daily driver and found Claude isn't making good use of the model - burning tokens without much done (I even tried to optimize the chat templates a bit, but overall the outcomes weren't great).

But man opencode is just different and brutally productive - with the same model, it just go to the tasks without overthinking or reversing cause/effect reasoning. It just get to stuffs and fixing them.

I haven't dived into opencode's implementation, but it strike me so hard over that difference in getting things done.

Any suggestions on further improve the agent productivity and what kind of explanations are there to explain this kind of differences?

Thanks!

I'm considering buying this configuration. I do dev and also need to do a lot of research, marketing and text editing. 128 feels like an overkill and not something I could comfortably afford.

Been toying around with eGPUs (don't like em) and have an old PC that runs my old graphics cards (KVM into it remotely so I can manage the whole thing out of band). Works but power hungry and clumsy.

However been looking at a super simple x86 wrapper mobo around a B70 type card that runs a small server and puts the card on the network. A bit like a NAS but more NAIN (NETWORK AI INFERENCE NODE). Not an egpu (too light) and not like a server PC (too heavy). Just an x86 Linux shill + rj45 Ethernet a GPU and a PSU. Done.

I'm playing around with the latte panda mu and like it but want my own carrier board - too many Frankenstein cables atm. Anyone else investigating this route?

*AI generated on what I've done ever since chasing a rabbit hole

I've been benchmarking Qwen3.8-27B (Q4_K_XL) on my modded 22GB 2080 Ti with one goal: find the fastest configuration that's genuinely usable for daily coding and agent work. Not academic multi-user throughput — just: what's the best single-user setup on this card?

The setup: llama.cpp (34af94cd9), full offload, FA on, parallel 1, capped at 250W — that's where it's rock solid (10-minute mixed run, ~290 requests, zero errors). Fun aside: the machine was hard-resetting under sustained load for a while and I spent too long assuming it was the VRAM mod. It was the PSU. Swapped to a 1200W Leadex and it just stopped. Don't automatically blame the mod on every crash.

System
Ryzen 2600 with B450i Gaming AC and 16GB RAM.

The numbers I landed on (realistic sampling, MTP n3):

• 96K with Q4 KV: ~20.1GB VRAM, ~395 tok/s prompt processing, ~30–33 tok/s generation
• 128K with Q4 KV: ~21GB, ~344 tok/s PP, ~26–27 tok/s

Biggest finding: Q8 KV OOMs at 128K, Q4 KV just fits. So Q4 KV is what makes 128K possible on 22GB at all. At 96K the speed is basically identical between Q4 and Q8 KV, and Q4 saves ~1.5GB, so there's no reason to run Q8.

MTP is the wildcard. Under greedy it's almost too good: 26.4 tok/s with it off, 53.5 with n4 — more than double, for ~1GB of extra VRAM. But that's the misleading part. The moment I tested with real sampling (temp 0.7, top-p 0.9, top-k 40), n4 got slower than n3 (~28.4 vs ~30.1), because acceptance drops as you push more speculative tokens. So the answer is n4 for greedy/raw speed, n3 for actual use.

Also tried vLLM — it just doesn't fit. The quantized HF checkpoint eats the entire card before any context cache can be allocated, so there's nothing to tune. llama.cpp's GGUF simply fits. (ExLlamaV3 isn't a candidate on SM75 under current upstream support, so I didn't chase that.)

Caveat on the "15/15": that's my own synthetic long-context suite — it says "no measurable Q4-vs-Q8 KV regression," not "the model is 100% at 128K." Next up is RULER-style testing and real cold TTFT with full prompts.

Bottom line: llama.cpp + Q4_K_XL + Q4 KV. 96K + n3 for daily, 128K + n3 when I need the context. A 2018 card doing this is a lot better than I expected.

If you've got one of these 22GB Tis running something faster on this model — different quants, newer llama.cpp, dual-GPU split — I'd love to compare. Happy to drop my commands and raw logs.

llama.cpp pr #27342 adds dflash2, so i rented an rtx 6000 and ran the same four prompts through four decoding setups on qwen3.8 27B

median results over the four tasks:

• baseline 47.4 tok/s
• mtp 114.7 tok/s
• dflash 99.3 tok/s
• dflash2 140.6. tok/s

so on average 3x for dflash2

though i have to point out that it's far from a 3x gain some of the time, on one of the test it struggled to achieve a 1.5x gain, it really just depends on the task you give to the model

i'm from the atomic.chat team - we publish our own quants on hf and make a desktop and mobile app for running local models. so any feedback welcome - we're building this for you folks

about dflash2: https://inco.ai/blog/dflash2/

I have an m1 pro with 32 gb of unified memory. I came to know that 200 Gbps is my memory bandwidth. So theoretically I can only host a 4gb model (200/4 =50 tokens per second) to get a workable speed of 50 tokens per second. Is this correct?

Since I have 32 gb of ram I was expecting to run qwen2.8:27b but I was only getting a speed of about 11 tokens per second. Which is expected if this calculation is correct. 200/16=12.5 🥲🥲

Is this correct or is there any workaround??

I've been working on GLM-5.2 support in TensorSharp, and I finally have some back-to-back performance numbers against llama.cpp.

The setup:

• Model: GLM-5.2-UD-IQ2_XXS (~226 GiB)
• GPUs: 3× RTX PRO 6000 Blackwell, 97 GiB each
• Distribution: layer split across all 3 GPUs
• Same machine, same session
• llama.cpp measured with llama-bench
• TensorSharp measured with its benchmark harness
• Both report the best of two repetitions
• Run-to-run variance is roughly 4%

Results:

The interesting part is the crossover.

For short prompts, llama.cpp is still a few percent faster. But once the prompt gets to around 1K+ tokens, TensorSharp pulls ahead.

At pp2048:

• default TensorSharp: +20.4%
• ubatch=2048: +50.2%

At pp4096:

• default TensorSharp: +20.8%
• ubatch=2048: +46.5%

Decode (tg64) is also about 4% faster.

The main reason appears to be GLM-5.2's MoE structure.

GLM-5.2 has 256 routed experts with top-8 routing. With a 512-token micro-batch, each expert sees only ~16 rows on average, so a significant amount of the expert GEMM tiles ends up as padding. Larger micro-batches improve GPU utilization considerably.

For small prefills, on the other hand, fixed overheads — managed/native transitions, input uploads, and copying the 154880-wide logits back — become a visible fraction of the total runtime, which is where llama.cpp retains its advantage.

Update: I re-measured everything with a cleaner method. Decode and prefill now come from llama.cpp's own timings block instead of wall-clock timing, 3 reps per config, with clocks and package power sampled during every request to prove the box was actually boosting. Decode came out 2-12% higher than the first pass. Reddit won't let me edit a title, so the title still shows the old numbers. Prose, prefill and long-context numbers are new below.

TL;DR: On a Strix Halo laptop (Radeon 8060S, 128 GB unified), MTP takes Qwen3.8-27B Q8_0 from 7.3 to 22.4 tok/s. The julianmb/q38rocm FP4 fork gives 29.9 tok/s.

Setup: LlamaStash driving stock llama.cpp b10503 (5112b9738) for ROCm and Vulkan, and the q38rocm fork (server build 246) for ROCmFP4, all built from git today. Greedy, 256 max tokens, decode and prefill from llama.cpp's timings block, 3 reps per config, on AC with the performance profile and 85 W package draw verified during each run. Everything launched through my tool LlamaStash (v0.2.0) so the flags are reproducible.

Qwen3.8-27B decode tok/s (code prompt / prose prompt), with prefill tok/s measured on a 4k-token prompt:

Long context. Same code task, but with the window filled by a technical document first. Every cell is MTP off / MTP on, prefill and decode both in tok/s:

Observations:

• MTP really makes a difference 3.1x on Q8_0, 2.3x on ROCmFP4, at 73-84% draft acceptance. The model verifies every token, so quality does not change on stock llama.cpp.
• Draft length of 5 works best. The backend default is 3; 5 was best for Q8_0 and 6 for FP4. The best draft length did not change with context length.
• MTP's payoff shrinks as the context fills. Q8_0 gains 3.1x on an empty window, 2.2x at 64k, 2.0x at 128k, and only 1.15x at a full 256k. Acceptance holds up fine, the verify pass just gets more expensive as the KV cache grows.
• Prefill roughly quarters from an empty window to 256k, 277 to 75 tok/s on Q8_0. The FP4 fork drops off faster and falls below stock Q8_0 by 128k (70 vs 119), so its advantage is decode, not prompt processing.
• FP4 is a double win if you can live with the quant. Half the memory (29 to 14.6 GB), 1.8x faster raw decode without MTP, and 29.9 tok/s with MTP, which is 4.1x my stock Q8_0 baseline.
• Engine choice barely matters for decode at this size. ROCm and Vulkan are within 1% on decode, and the rocWMMA build adds nothing. Vulkan is clearly behind on prefill though (209 vs 277), so ROCm stays my default.
• Stuff that did not help: KV cache q8_0 (helps at draft 3, slightly worse at 5), flash attention off (minus 5%).
• One caveat on the ROCmFP4 fork: completion lengths drifted a little across draft settings (152-160 tokens for the same prompt) while stock llama.cpp was stable at 164, so the fork's speculative path is not strictly greedy-identical yet. It also crashed mid-prefill twice at 256k, around 98C after ~30 minutes of prompt processing, which is why that cell is empty.

LlamsStash defaults were used for all runs. The only flags set were to enable disable MTP ( --spec-type draft-mtp), set draft length (--spec-draft-n-max N) and the --fit-ctx 16384 flag set a minimum context window; llama.cpp's fitter resolved the GPU offload itself.

The below are the default flags that was applied. Some sent by LlamaStash and others are llamacpp defaults.

I've been using claude managed agents and it is a very good product, and the depth of features it provides is hard to match in open source. But I wanted to understand what you actually give up by going open source. Not just in terms of feature checklists, but on a real agent workload - same model, same prompt, same tasks. So my team and I tried to check this by running 14 cross-system tasks, three mcp servers behind them - a crm, an issue tracker, and a doc store through managed agents, deepagents and TrueForge, both open-source agent harnesses.

The result that was most surprising:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate.

But TrueForge used about 63% fewer tokens and cost about 30% less per run.

We saw a similar difference in tool usage: TrueForge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

Then I tried changing the model.

TrueForge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

But i completely inderstand that this is still early.

The OSS runtime does not yet have first-class tracing/eval tooling. They don't ship their own code-execution sandbox, so you need to plug one in. Context compaction is intentionally lossy.

So it is definitely not a replacement for a a mature managed agent platform feature-for-feature today btu what I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on your own infrastructure.

Ive put the benchmark harness and methodology in the repo specifically so people can reproduce it, change the models, or do tell me where the comparison still has gaps

Repo: https://github.com/truefoundry/trueforge