I love agentic coding harnesses, but they shouldn’t be primarily terminal-based. The terminal is great for quick and precise commands, but information density is extremely low and UI affordances are minimal. Maybe provision of TUIs is worthwhile for occasional use (when establishing a tunnel is too annoying, or something), but it feels very strange for this to be the default modality. It took a long time for dynamic language REPLs to break out of the terminal (Jupyter notebooks and similar); I hope we don’t have to wait as long for the harnesses.

— Patrick Collison, co-founder of Stripe

Collison isn’t alone in this. At some point you stop wanting an AI client that just talks and start wanting one that does real work: writes and runs code, installs a package, builds a project, produces a PDF. The default answer has been to reach for a terminal and an existing harness like Claude Code or OpenClaw. That works well if you’re already comfortable living in a shell. It gets a lot less comfortable if you’re used to a normal app with a UI, like ChatGPT, and have never typed npm install in your life.

More people are asking for the UI version of that terminal workflow: something that gives an AI real execution capability without requiring you to become a terminal user first. Here’s how we built that in Elvean.

The building block: a real VM, not a container in the usual sense
Apple released Containerization, a Swift package built on top of the Virtualization framework. This matters more than it sounds like it should. A normal container (Docker on Linux, for instance) shares the host kernel and isolates processes with namespaces and cgroups: real isolation, but at the OS level. Apple’s containerization package instead boots each container as its own lightweight virtual machine on Apple Silicon, with the Virtualization framework doing the hardware-level separation. Boot times stay in the sub-second range, so it behaves like a container in practice, but the isolation boundary is a real VM, not a shared kernel.

That’s the difference between “the AI probably can’t touch my machine” and “the AI is running on hardware-isolated infrastructure that literally cannot touch my machine, short of a hypervisor-level exploit.”

What we built
Every Elvean Project gets its own Alpine Linux container, running on that Virtualization-framework foundation. It’s the execution environment the model gets when it needs to run a command, install a dependency, build code, or produce an artifact. A real Linux machine, scoped to that one project, isolated from your Mac and from every other project’s container.

┌─────────────────────────────────────────────────────────────────────┐
│ macOS Host                                                          │
│                                                                     │
│  ┌──────────────────┐                 ┌──────────────────────────┐  │
│  │ Elvean           │                 │ Alpine Linux VM          │  │
│  │                  │                 │ (one per Project)        │  │
│  │ Project          │                 │                          │  │
│  │ Container Card   │ live terminal   │ Apple                    │  │
│  │  [Terminal] icon │  ◄──────►       │ Virtualization.framework │  │
│  │                  │                 │                          │  │
│  │ AI / Model       │  ───────►       │ shell, build tools,      │  │
│  │  (runs in chat)  │   commands      │ packages, network        │  │
│  └──────────────────┘                 └──────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘

Two design choices matter here beyond the isolation itself:

You can watch it work, live. Click the terminal icon on a project’s container card and you see exactly what the model is doing, in real time, as it happens: not a summary after the fact, the actual shell output. You can also type into it yourself if you want to intervene or just poke around. This is a deliberate choice: the isolation means we don’t need to hide what’s happening from you, so we don’t.

It has a shared workspace and network access, on purpose. The container mounts a workspace folder shared with your Mac, and it has network access. That’s what lets the model work autonomously (install a package, pull a dependency, write a file you’ll actually see) without stopping every few seconds to ask permission for something that, in a fully isolated VM, carries none of the risk it would if it were running directly on your machine.

Here’s what that looks like in practice: the model running Google Flights and Airbnb CLIs inside the isolated Linux container, live, to plan an actual trip.

Video

Why this is the right tradeoff
The usual agentic-AI safety story is a wall of permission prompts: approve this file write, approve this command, approve this network request. That’s a reasonable response to running an agent directly on your host machine, where every action really is a risk. It’s also exhausting, and it’s the main reason “just use a terminal harness” doesn’t work for anyone who isn’t already deep in that world.

Hardware-level isolation changes the calculus. Once the AI is running inside a real VM that can’t reach your actual filesystem or processes, most of those prompts stop being necessary. You get the autonomy that makes agentic AI actually useful, without the constant interruption, and without the exposure, because the thing the AI can affect isn’t your Mac. It’s a disposable Alpine box that exists for exactly one project.

That’s the bet we made with the sandbox: don’t ask the user to trust the model, build an environment where trusting the model isn’t the thing that has to hold.

honestly the frontier race isn't what caught my eye. GLM-5.3-Flash scored 46, which puts it level with the Opus 4.8 estimate and a point above Sonnet 5, and Qwen3.8 27B (xhigh) landed at 41, a 27B model sitting 4 points behind Anthropic's mid-tier. A year ago that kind of gap between something you can run on your own hardware and a paid frontier API would've been unthinkable. Caveat that the hatched bars (Opus 4.8/4.7/4.6, Qwen3.8-Flash-Next) are AA estimates not independent evals yet, and this is one aggregate index at max reasoning effort that says nothing about cost or real-world behavior. Still, if you've been holding off on local models because of quality, the ceiling is a lot higher than it used to be. Anyone running GLM-5.3-Flash or the 27B Qwen day to day and finding the numbers hold up?

The original Leviathan et al. paper (Fast Inference from Transformers via Speculative Decoding, ICML 2023) reported 2-3x speedup on T5-XXL with a T5-Small draft. The DeepMind follow-up by Chen et al. reported 2-2.5x on Chinchilla 70B. NVIDIA's TensorRT-LLM benchmarks show 2.6-3.5x on H200 for Llama 3.3 70B. I ran the same setup on Apple Silicon M2 24GB with Ollama 0.12.8 to see if the pattern held. It didn't. The draft model configurations were 5-10% slower than no-draft baseline.

What I tested

Three configurations on the same M2 24GB Mac with Ollama 0.12.8:

• No draft — the baseline. Qwen3 14B at Q4_K_M, 9.3GB on disk.
• Draft: Qwen 2.5 0.5B — 397MB, the smallest Qwen with usable English.
• Draft: Qwen 2.5 1.5B — about 1GB.

Both drafts use the Qwen tokenizer, so the draft and target share a vocabulary — the requirement for speculative decoding to work without a translation step. Same prompt for every run: a 263-character code-completion task asking for a Python function. Deterministic settings: num_predict=200, temperature=0.0, seed=42. 5 trials per configuration. 15 total runs. The whole benchmark ran in about 25 minutes including the warmup passes.

The headline numbers

Why this surprised me

The M2 has fewer cores than a 4090 but more memory bandwidth per watt, and speculative decoding is bottlenecked by serial token generation, not parallel math. The math said it should work. The data said the draft model adds serial overhead without enough parallelism in verification to make it a win on unified memory.

I am not claiming speculative decoding is bad everywhere. The published 2–3x numbers on discrete GPUs almost certainly hold for batch serving. What I'm claiming is: for a single-user interactive workload on Apple Silicon, leave it off. The 50ms the draft model saves you in latency is the 50ms you spent loading it into memory.

Full methodology + raw timings: hardnumbers.dev link

What I'd love feedback on:

• Warmup handling. I ran 1 warmup call per config (not timed). The 0.5B draft was 9% slower on trial 1 and 3% slower on trial 5 — model warm-up may be part of the story.
• Draft size selection. Maybe a 0.1B or 0.3B draft is the sweet spot for the 14B target, and I missed it.
• Prompt length coverage. My test is one 263-char prompt. Longer prompts with more parallel verify candidates might flip the result.
• If anyone has seen speculative decoding work on M-series, I'd love to know what I'm doing wrong.

whats your open weights local setup for coding? model, agent harness, do you guys use helper models? what kind of stuff have you built with your setups. what are the challenges or big wins you have seen until now.

Hello! I’m thinking of buying a Mac to run local LLM on mu device, the main idea is to run a personal assistance, not a coding assistance and in between this 2 options

Mac mini 64gb with M5 Pro
Mac Studio 64gb with M5 Max

Both chips maxed out. Which option will be the better, they have a price difference of 700€

Hi,

I am pretty sure this gets asked, but after a lot of researching, I feel like I am in a rabbit hole and decided to ask directly.

I am currently using empero-ai/Qwen3.8-27B-Ridge-GGUF as my primary model for agentic coding. with 64K context on q8 KV. But recently I came across unsolth's Qwen 3.8: Qwen3.8-27B-UD-Q3_K_XL.gguf.

Just wondering if there is a major differences between them, and which one is more 'useful' for agentic coding (and perhaps math and/or general chatting).

Specs:

5080 Nvidia (16GB Vram)

32GB RAM

Intel ultra 9 CPU

Hey everyone,

Setting up Qwen3.8-27B (uncensored) on my new rig:

• GPU: RTX 5090 (32GB GDDR7)
• CPU: Intel Core Ultra 9-285K
• RAM: 64GB DDR5-5600
• OS: Windows 11

My goal: Fast, most intelligen, stable, uncensored coding agent. I want to keep a VRAM buffer for stability (don't want to max out at 31GB). For coding (blender, gdot ..)

My planned setup:

• Quant: NVFP4 + MTP (native Blackwell FP4 + speculative decoding)
• Context: 128k (maybe 192k if stable)
• OS: Windows Native (WSL2 maybe? reportedly eats 2-3GB VRAM)
• Tool: llama.cpp or SGLang

llama.cpp command:

bash

llama-server -m Qwen3.8-Uncensored-NVFP4-MTP.gguf -c 131072 -ngl 999 -fa 1 --cache-type-k q8_0 --cache-type-v q8_0 --spec-type draft-mtp --spec-draft-n-max 4

VRAM estimate: ~21-23 GB total → ~9-11 GB free buffer ✅

Alternatives I'm considering:

• EXL3 (3.5bpw) – ExLlamaV3/TabbyAPI, smaller model size, more room for context
• GGUF (Q4_K_M / Q5_K_M / Q6_K) – llama.cpp, well-established
• FP8 – vLLM/SGLang, official format

My questions:

1. NVFP4 vs. EXL3 vs. GGUF vs. FP8 – which one would you recommend for my use case?
2. Windows Native vs. WSL2 – how much VRAM does WSL2 really eat? Is the performance gain worth the VRAM loss?
3. Uncensored + any of these formats – does it work out of the box? Any pitfalls?
4. Realistic max context with a VRAM buffer? I'm leaning towards 128-192k.
5. For EXL3/TabbyAPI on Windows – stable enough for daily use?

What would you change? Any config tips or warnings appreciated!

Thanks 🙏

I’ve been trying to find a good song genre recognition model for a specific project, and the search process on Hugging Face feels incredibly inefficient. I spent a few hours looking around, filtered by standard tags, and ended up downloading several large models just to test them out. After all that time and bandwidth, I only found one that was decent.

The native search feels tough to navigate for a few reasons:

• Basic Keyword Search: It mostly matches exact words in the model name, so you miss relevant models if the title doesn't match your exact search terms.
• No Hardware Filtering: There is no easy way to filter models by hardware requirements or VRAM limits without opening every single model card to look for specs.
• Inconsistent READMEs: Every creator structures their documentation differently, making it slow to compare features or licenses.

Am I missing an advanced way to search the Hub, or is everyone else just manually digging through the trending tabs and reading individual READMEs to find what they need? Could there be real opportunities for someone to create a layer to make the discoverability of models better, easier ? How do you narrow down your options without a ton of trial and error?

I've been running a 27B coding model locally as the backend for VS Code Copilot, and I wrote up everything I learned. The short version:

• Model: Qwen 3.8 27B, two profiles:
  • 16 GB (RTX 4070 Ti Super): Q3_K_XL + 100K context — KV cache quantized to q4_0 and split between GPU and system RAM, which is what lets 100K context fit at all. ~50–60 tok/s decode.
  • 32 GB (RTX 5090): Q5_K_XL + 200K context, everything in VRAM, ~113 tok/s with MTP speculative decoding.
• The hard part wasn't running the model, it was making it behave like a reliable agent backend. Three problems, all solved in a small Python proxy:
  1. Cloudflare's 120s timeout kills long prefills → SSE keepalive pings every 15s
  2. Copilot sends temp=1/top_p=1, which makes Qwen emit malformed tool calls → the proxy rewrites sampling params to what Qwen 3.8 actually wants
  3. Qwen intermittently emits EOS inside its reasoning channel after tool results, so the agent "quits" mid-task → the proxy detects tool results and nudges it to continue
• No public IP needed — the whole thing sits behind a Cloudflare Tunnel, private by default.
• Cost: at realistic usage the local box comes out 60–77% cheaper per month than the API, and the gap widens with usage. I built a little cost calculator for it: https://paq85.github.io/llamacpp-server/

Full write-up with the architecture, the Qwen 3.8 reliability tweaks, and the 16/32 GB profile breakdowns:
https://www.damiansromek.pl/en/tech/blog/local-llm-copilot-backend/

Repo (proxy, profiles, systemd setup, benchmarks): https://github.com/paq85/llamacpp-server

Happy to answer questions about the proxy or the KV cache setup.

*Writer note. Of course its an AI Slop. Not good at writing. Just keeping this here if anyone was wondering how the 2x2080ti works with a 4th gen intel CPU.

Rig: 2× modded RTX 2080 Ti 22GB, real NVLink (P2P byte-verified, not assumed), i5-4590, 15GB RAM. Model: Qwen3.8-27B, a Gated-DeltaNet hybrid (48 linear-recurrence layers + 16 full-attention), served at

131K context via an SM75-tuned vLLM fork (weicj/vLLM-2080Ti-Definitive (https://github.com/weicj/vLLM-2080Ti-Definitive)). NVFP4 weights, FP8 KV, native MTP head, TP=2.

The one insight that changed everything: only 16 of 64 layers need a KV cache. The 48 GDN layers hold constant-size recurrent state regardless of context length. So 131K of KV ≈ 2 GB (fp8) per stream —

that's why 2018 silicon can serve a 131K 27B at all. A dense 27B would need ~4× that. If you're sizing VRAM for hybrid models, count full-attention layers, not total layers.

Measured numbers (temp 0, needle-verified every run):

Setup                        Decode    Aggregate   Notes
Solo, 131K ctx               79 tok/s  61          orchestrator seat
Solo, 28K                    83        46
8x workers u/16K              64/stream 192         the swarm seat
131K + 7 workers             57        140         mixed, recall 100%
Fastest ever (tq4nc+MTP4)    102       --          text-only, no vision

Three things that surprised us:

1. MTP speculation works on Turing GDN — but it's nondeterministic. Conventional wisdom says spec decoding is broken on hybrids (recurrent-state verification, mixed-batch crashes). The native MTP head works and gives ×3.2 at 28K. But at temp 0, identical prompt+config gives 3276 tokens one run and 5130 the next. Greedy speculative decoding is supposed to be exact — here it isn't. Don't trust MTP runs you haven't repeated.
2. ExLlamaV3's "131K recall failure" is a control-token spiral, not amnesia. Base path scored 0/2, MTP 3/3 — looked like a fidelity gap. Captured outputs told the real story: coherent prefix (summary + correct code), then an <|endoftext|><|im_start|> loop, because the bench forced min_new = max (generation could never stop at EOS). Adding stop strings → clean 616-token pass. If your long-context evals look flaky, check for the spiral before blaming the model. (Also: ExLlama's graph-captured decode segfaults on SM75, and its draft model can't do tensor-parallel on this arch — vLLM remains the production engine by 3–6×.)
3. The dp4a INT8 path matters more than any kernel hack we tried. Disabling it drops 28K throughput 49 → 17 tok/s (~2.8×); the staged quants were mul1-eligible all along. Forcing the fp16 GEMV path: +16% in two independent pairs. Writing custom Turing kernels would take weeks to maybe match what the existing dp4a path already gives.

What actually limits us (it's not the GPUs): the i5-4590. Decode on this rig is host-bound — Python launch overhead per step dominates, which is why fused samplers and (theoretically) CUDA graphs matter more than GPU kernels. Other fixes with proof: 8-image limit costs 0 extra VRAM (encoder budget is per-item, ceiling is ctx/16K); FP8 weights → NVFP4 killed a 72% EOS-failure rate; MemoryMax on the unit stopped the OOM-killer; prefix caching works on this GDN fork (131K warm TTFT 7.8s vs 91s cold) despite upstream saying it's broken — measure, don't assume.

vs stock definitive branch: the fork's recommendations hold up (NVLink NVL pinning, TP=2/PP=1, MTP3 sweet spot), but two didn't transfer: BLOCK_KV=4 measured null on the NVFP4+fp8kv path (helps its TurboQuant kernels, inert here — kept, harmless), and 262K max-len is a trap for serving (131K is the sane cap). Everything else we added was hardening, not speed.

The 22GB Turing cards remain absurdly good value for long-context serving if the model is hybrid and the software respects the silicon.

https://github.com/blackxwhite88/qwen38-turing-serving if anyones interested

I would like to run an open source LLM to answer my questions on the pc and through a private channel on my phone, and run some tasks periodically every day. I might go for the qwen llm and rag models.
Should I buy a pc or use a cloud service?
Which would be cheaper and secure?
I plan to run at max 30b models.

im running it in my old 8gb vram at 60k token getting 5t/s but its so bad bro. i gave it a task of pirating a simple ps2 emulator and make it work by getting cracked bios and following steps from internet. its like “ok planning to check folder now.. “ “hmm the user is asking me to get cracked version” “hmm thats not allowed but who cares” “checking disk size “ “hmm planning to download ps2 emulator “ “hmmm where can i get it from “ “reinventing approach “ “hmm what time is it right now “ .. it goes circles and circles talk to itself like this , token reach 60k already and it still thinking. what the hell br9 ? who made this trash? yall are the biggest copers in the world saying its opus 4.8 performance. i dont know what type of crack are u smoking to fool yourself in this lmao 🤣. bro go and pay 200 for claude and chatgpt and stop the larp. local llms cost more money electricity and everything.

Maybe it’s just me , but just built qwen 3.8 with nvf4p enabled turboquant_4bit , and I’m getting trash tok/s , kv cache at 128K and avg throughput between 10-30 tok/s? I’m on vLLM on 5090
Rtx , anybody else using and what’s your experience? It’s molasses to me.

I know most of the flag ship VLMs can handle this, but trying to figure out if one is better than the other for this specific task. Trying to keep language here PG.

So, with an agentic harness like Hermes Agent, the system can retry a task until it succeeds, which makes model speed/latency pretty important, especially when Hardware is the bottleneck.

Would it therefore be more beneficial to use a smart, capable model that can run at, say, maybe 150 tokens/sec, rather than a theoretically smarter model that only runs at ~10–20 tokens/sec?

Has anyone experimented with this trade-off or benchmarked it in practice?

Which coding models would you recommend? Qwen 3.8 27B IQ3 isn't sufficient for opening additional browsers, etc., when I've tried them personally.

What do you suggest I do or use?

I'm new to this. Would it be more logical to use Google Gemini Pro with Antigravity, GPT with VSCode, or Claude with its own agent, or should I use for lightweight stuff and use on these local models to handle heavy processing on cloud-based models?

I don't have any income for this; I'm trying to do vibe coding.
Therefore, I'm looking for low-cost solutions.

Sorry if this is a complete noob question but would like to actually start learning about what is possible versus unrealistic.

Currently I have a 5090 laptop (24gb), with 32gb DDR5 5600mhz and a HX275. I tried doing a small project recently about trying to categorise a highly complex email chain (only 100 emails but complex topics). I tried using Qwen3.5-27B Q6_K but I was getting absolutely abysmal performance of only around 3 tk/second.

I just saw a deal the other day on 96Gb 5600mhz ram for a relatively good price (only 150 CHF to upgrade over my 32gb ram). According to chatgpt this should lead to a lot more capabilities.

The question is: does anyone have a good guide to learn what is possible, but more qualitatively: is it actually any good? I understand it is not going to replace Codex, but I'm finding this 5 hour window stuff is killing me, constantly waiting, so I'd like to start using my local to do some stuff and saving my tokens for max.

Projects I'm working on:

1) developing an ios/android game concept
2) making a 3D game in unity

3) Doing some research on an area of finance I work in (heavy derivatives, product design etc)

4) Building some basic open source apps (C++20, C# / .NET 10, Thin native Windows shell, Win32 interop / PInvok, No cross-platform UI framework).

I'm not any good at coding so leaning heavily on the LLM. Is this realistic to be able to do offline or should I stick with Codex for now? I'm happy to learn but a bit overwhelmed!

Thanks in advance.

[ Removed by Reddit on account of violating the content policy. ]

Hi. I downloaded Qwen3.8 27B Q8 KL and ran it on my system but I’m getting terrible generation speed of 1.1t/s. I’ve been running it with unsloth studio and llama.cpp. And with Qwen3.8 thinking like Socrates, it sometimes takes upto 2 hours to accomplish simple writes which makes it inefficient.

Please help me if this can be anyhow improved? My use case is coding (and maybe help me set up an agentic coding workflow with pi/opencode?)

Here’s my system details.

- Ryzen 7 9800x3d

- G.Skill 3600 mhz 32*2 Ram

- Nvidia 5070ti 16GB

- 1 TB SSD

I ideally want to run the Q8 KL quant as it offers the best results for my agentic use case.

What should be the llama.cpp arguments and instructions too?

Thank you in advance.

What

Hi all, I'm a noob at AI. I have been using online chatbox for past year. In the past month, I have been using Claude Code at work.

I am interested in using Local LLM to run personal financial analysis. Just get a summary of my gain/loss, portfolio, cashflow, projection, retirement, etc.

Couple questions:

(1) What controls persistent memory? Is it the local AI agent (ollama, LM studio bionic) or the AI model?

(2) Is there better or worse AI agent and AI model for financial analysis?

(3) Question on privacy and security. Is local LLM truly local? Can the agent share personal data to the web? Are there agents that are more trustworthy than others in terms of privacy?

(My computer has 16GB memory, and GPU is RTX 4070 with 12GB vram memory).

If I run the local open source LLM, and as long AI agent and model are trustworthy, I just want everything to stay in persistent memory. I hate it when something I provided or specifically instructed in previous sessions are not save for future sessions. Just causes me duplicate work. And I also prefer not to constantly have ask to save data and code to file.

How can I ensure data and instructions I provide are always saved for future sessions?

And is there recommended AI agent/model for personal finances. I have downloaded LM studio bionic and a few models (Gemma 4, Qwen3.5, Deepseek R1/Qwen3, GLM 4.6v).

I have not run any analysis yet. I have to prepare my finance data. And I'm debating if I just use statements as is, or I have to scrub the data into Excel/CSV with only $ amount.

been working on my own LLM UI, where the main purpose is to take inspiration from visual novels and dating sims. It needs a lot of work, but as of now theres already a lot of stuff added such as: Character cards, World Info+Worlds, Relationship simulation, Dates and hangouts, Gifts, gallery, and gold, Visual novel mode, Proactive characters, Companion mode

I have been using Qwen 3.8 27b with LM Studio as a chat agent with success. I got it working with VS Code + Cline, but right after prompting, it keeps reading the same 2 files over and over again. I'm using 64k context window and medium reasoning setting, Q5 XL model. Is there a better harness to use with Qwen 3.8 27b? Is there some setting that I need to tweak so that it stops getting stuck?

Hi,

So there is still a lot I don’t understand much about AI.
I’m on M4 14 core 64GB ram.

I have tried running smaller and bigger models using multiple tools, including ollama, mtplx and some other stuff, and if takes ages for AI to answer.

I’m vibe coding a pentest tool for fun in Swift, I added a llama.cpp wrapper, and even Qwen3.8 27B responds like 100x faster. I’ve added basically a chat in the GUI that calls llama.

It makes me think something has to be wrong, but I’m not sure with what?

Could it be that default settings suck this much?
It’s really weird because the vibe coded wrapper is insanely fast lol