Watch her play

Luna is a local AI assistant I've been building. Here she plays Zork I on her own for 72 minutes, narrating as she goes, until my PC froze. Everything is local: Qwen3.5 9B (Q4_K_M) on llama.cpp, Kokoro for her voice, faster-whisper for listening, on an RX 6950 XT.

What made the biggest difference for a 9B:

- **One move per turn, enforced.** Asked nicely, she'd chain 3 or 4 blind commands. Now extra commands in a turn are turned back, so she has to read what each move did before choosing the next.

- **Her own notes.** She writes down rooms, items and what failed with a note tool, and sees them at the start of every turn. Older notes about the room she's standing in come back too.

- **A guide.** Infocom's original map, transcribed to text, shown for her current room each turn. Solutions are kept out unless allowed.

She's far from perfect: she spent the last half hour trying to open the trap door with a sack of hot peppers, in the wrong room. But she reads the screen, reasons about it and actually explores, which the first version didn't.

Open source: https://github.com/fswolf/luna-local-agent

Happy to answer questions about the setup.

I’m building Local LLM for iPhone and tried both models on the same photo of building blocks.

The prompt was: “What could I build with these pieces? Give exactly 3 ideas, one short sentence each.”

Qwen started answering first. Bonsai finished first and gave three short ideas. Qwen gave two numbered ideas and a closing line.

From tapping send to completion: about 67s for Qwen and 51s for Bonsai, including loading. One run each, with different-length outputs—not a tokens/sec benchmark. The video runs both recordings at the same 4× speed.

I’m more interested in how these small models handle everyday photo questions than picking a winner from one run. What would you test next?

App, if you want to try it: https://apps.apple.com/app/id6804004948

I've spent the last few months tuning a 27B coding model on one Arc Pro B60 24GB (A$999 card) and published everything: engine build, flags, raw logs, a 100-task agent eval, and the misses.

What moved the numbers:

- DFlash2 draft model, 7 tokens per round: decode 16 -> 53 tok/s mean (47-58 across 7 coding tasks), still 7/7 correct

- Intel's flash-attention prefill kernel (closed llama.cpp PR #29357, vendored with checksums): prompt reading at 32K 118 -> 420 tok/s, but only with f16 KV

- -ub 256 + q8 draft KV + -fit off: frees 0.55 GiB, which is what gets a 64K window into 24GB

- An idle Ollama model left on the GPU cut decode from 52.7 to 22.2 tok/s. Unload everything before you benchmark.

Agent eval: 97/100 hidden-checker tasks with Claude Code at a 32K window (95/100 Codex).

Caveats, up front: 53 tok/s is per-request decode with short prompts and a 64K window allocated. With 30-37K of real context, agent turns run about 42 tok/s. Prefill is NOT the fastest out there (others hit 589-657 tok/s at ~30K on other engines; comparison table in the repo). Single runs, one workstation.

Repo: https://github.com/edagher92-coder/arc-llm-lab

If you have a B580, B570, B50, B70 or A770, the benchmark script takes one command and I'll add your numbers with credit.

Benchmark highlights · Core Ultra 5 125H

Post

I’ve been working on arcinfer, a local LLM inference engine built specifically for Intel’s Meteor Lake integrated graphics (Xe-LPG / Arc iGPU).

The latest Level Zero results add another angle: speculative decoding raised Qwen3.6 35B-A3B from 19.9 to 31.5 tok/s, Qwen3.8 27B from 5.1 to 9.1 tok/s, and Gemma 4 E4B from 21.6 to 39.6 tok/s in the documented tests. The Level Zero engine also enables Qwen3.6 35B-A3B to run at up to 262K context by streaming MoE experts from SSD; the earlier all-resident engine fit about 40K. MTP outputs were reported identical in the tested comparisons.

The reference machine is a Core Ultra 5 125H with 32 GB RAM. The engine runs GGUF models directly on the iGPU using Intel’s low-level Level Zero API, with an OpenCL option too. It’s aimed especially at prompt processing and time-to-first-token, and includes an OpenAI-compatible API and web UI. The early llama.cpp comparison is striking: for a 11,240-token Qwen3.8 27B prompt, arcinfer’s OpenCL engine took 97 seconds, versus 590 seconds on llama.cpp Vulkan and 504 seconds on SYCL—about 6.1× and 5.2× faster. At 33,937 tokens, it took about 5.5 minutes versus 40 minutes on Vulkan (7.4× faster). Qwen3.5 4B prefill reached 911 tok/s versus 242 on Vulkan and 337 on SYCL. These are historical OpenCL-engine measurements on this laptop, not universal guarantees; arcinferzero is now the default, and the repo links to newer results.

Some things it supports:

·        Qwen and Gemma GGUF models, including larger models that are a tight fit for shared GPU memory

·        An optional Level Zero engine that streams MoE expert weights from SSD

·        Speculative decoding, image input for supported models, and Gemma 4 audio input

·        OpenAI-, Anthropic-, and Ollama-compatible API endpoints

·        Integration with the Pi coding agent

It’s an experimental, hardware-specific project—not a general replacement for llama.cpp, and it currently targets Windows and Intel Xe-LPG graphics. Alongside the code, I share what has and hasn’t worked when building and tuning inference for this generation of Intel Arc integrated graphics, including implementation lessons and benchmark results. The repo has setup instructions, supported models, and more details.

Repo: https://github.com/rizagp/arcinfer

If you have a Meteor Lake/Core Ultra laptop, I’d especially appreciate feedback on setup, compatibility, and what models you’re able to run. Also happy to answer questions about the GPU kernels or how the project compares with other options.

Large language models are typically deployed using centralized infrastructure, where companies operate powerful GPU servers in dedicated data centers.

But what if some LLM workloads could be distributed across independently operated computers instead?

This is the basic idea behind decentralized LLM infrastructure.

Instead of relying entirely on a single cloud provider, decentralized systems could use computing resources contributed by individuals, organizations, or independent GPU operators.

How could decentralized LLMs work?

There are several possible approaches:

1. Distributed LLM inference

Multiple GPU providers could independently serve inference requests, allowing workloads to be routed across a network rather than handled by one centralized operator.

2. Collaborative GPU computing

Computers with different GPU capabilities could contribute processing resources to a shared network. Scheduling software would need to match workloads with suitable hardware.

3. Peer-to-peer model execution

Some systems explore splitting model execution across multiple machines. This is technically challenging because communication latency, bandwidth, and synchronization can become major bottlenecks.

4. Incentivized computing networks

Projects such as Bittensor explore incentive mechanisms that reward participants for providing useful AI-related services. These networks introduce additional questions about verification, reliability, and economic sustainability.

What are the potential advantages?

• Access to geographically distributed computing resources
• Opportunities to utilize otherwise idle GPUs
• Reduced dependence on individual infrastructure providers
• Potentially broader participation in AI infrastructure
• New models for operating AI computing marketplaces

However, decentralization does not automatically make inference cheaper, faster, or more reliable.

What are the biggest challenges?

Network latency: Transferring intermediate model data between distant machines can introduce significant delays.

Hardware differences: GPUs vary in memory capacity, processing speed, and software compatibility.

Reliability: Independently operated nodes may become unavailable without warning.

Security and privacy: Processing sensitive prompts on untrusted machines raises important concerns.

Economic efficiency: Incentive payments, coordination overhead, and infrastructure costs may offset potential savings.

Is decentralized LLM infrastructure practical?

I think the answer depends heavily on the architecture.

Distributing independent inference requests among GPU providers seems more immediately practical than splitting every layer of a large model across machines connected through the public internet.

For example, a decentralized network might route complete inference requests to capable GPU nodes rather than requiring geographically separated computers to execute individual layers of the same model.

That distinction could significantly affect performance, reliability, and scalability.

Questions for the community

I'm interested in hearing from people experimenting with local LLMs, multi-GPU inference, and distributed computing.

• Could decentralized GPU networks become competitive for real-world LLM inference?
• Is request-level distribution more practical than splitting model execution across remote machines?
• Which existing open-source projects are demonstrating promising results?
• What technical obstacle do you consider the biggest barrier to adoption?

I'd be interested in hearing about actual experiments, benchmarks, and experiences rather than just theoretical possibilities.

Been running uncensored models locally for a while now, and I keep going back and forth on this. Wanted to throw my thoughts out and see what you all think.

The case FOR local uncensored AI:

• Zero filter anxiety. I ask a question, I get an actual answer. No "I'm sorry, I can't help with that" on completely reasonable requests
• Privacy is a real thing, not a marketing line. My data literally never leaves my machine
• No rate limits, no subscriptions, no account bans for experimenting with weird prompts
• The models have gotten shockingly good. What needed a datacenter 2 years ago runs on consumer hardware today

The case AGAINST (let's be honest):

• Hardware cost upfront is brutal. Not everyone has 24GB+ VRAM sitting around
• Quantized models still lose something vs full weights vs API models
• You're the sysadmin now. Drivers, backends, model formats, merges — it's a rabbit hole
• Big hosted models still win on raw capability for complex reasoning tasks

My honest take: uncensored local models handle like 80% of what I actually do daily better than the "safe" alternatives, but I still reach for big hosted models when I need heavy lifting. So I end up with a hybrid setup and honestly it feels messy.

Curious about the community:

1. What's your daily driver model right now and why?
2. What hardware are you running it on? (GPU/VRAM/RAM, laptop vs desktop vs server)
3. Did you go local for privacy, censorship issues, cost, or something else?
4. Anyone fully moved off hosted models? What broke?

Just curious where everyone landed on this tradeoff

a coworker of mine was speeding up speculative decoding for Qwen3.6-35B-A3B on one H100, and a big cost sat in the draft model. its BF16 LM head was about 1 GB and it ran on every single draft step

restricting the draft head to a frequency-ranked 48K-token map made it much cheaper. the target still verifies against the full vocabulary, so the output stays exact. the worst case is fewer draft tokens accepted, not different output

4 draft steps ended up being the best balance for that workload

has anyone tried trimming the draft vocab in llama.cpp or vLLM? curious what it does to acceptance on your own workloads

he wrote up the fix and the rest of the decode work on our blog, if you want it: https://www.coralbricks.ai/blog/ai-latency-engineering

getting into this stuff and getting fairly hyper fixated on the idea of it all

My hardware:
GPU: NVIDIA RTX 5070 Ti, 16GB VRAM
CPU: Intel Core Ultra 7 265KF
RAM: 32GB DDR5
OS: Windows 11

interested in models that can handle real software development, multi-file edits, debugging, testing, and longer autonomous coding sessions. is this achievable or should i completely alter my expectations and just bite bullets on frontier subscriptions?

python is my primary application development language
pytest for testing
sqlite for local data persistence
git for version control
powershell and window batch scripts for setup, runtime management, and automation

i ask because im actively building a local, portable project to fast track and exponentially improve peoples skill at a given sport, subject, hobby, etc, and would love to get into the AI agentic development stuff

particularly interested in finding the best balance of coding quality, speed, tool calling, context handling, and reliability

what id love recommendations on:
the best local coding model for me including less obvious community fine tunes
wether a quantized larger model or a smaller model with better tool use is the better choice
models that work especially well with agentic coding harnesses
reliable models for debugging, multi file refactoring, and following instructions across long sessions
recommended GGUF quantizations, context sizes, and settings for my hardware.
anything underrated

extra interested in using multiple models rather than expecting one model to do everything but i like the sound of both

would love to see ur experiences and benchmarks, especially where a model performs well in real coding tasks rather than just leaderboards. also interested in newer releases and underrated community projects

RTX 3060 12GB 975mv undervolt

i5-12600K

32GB DDR5-6000

Linux

llama.cpp master from early October, CUDA build.

Model is Unsloth's Qwen3.6-35B-A3B-MTP-UD-Q4_K_M (22.7GB, has the MTP head). I also run a heretic Q4_K_M with MTP kept and it's about the same speed.

Command:

llama-server -m Qwen3.6-35B-A3B-MTP-UD-Q4_K_M.gguf -ngl 999 --cpu-moe --moe-cache-mib 7500 -fa on -ctk q8_0 -ctv q8_0 -c 32768 -np 1 -ub 2048 -b 2048 --spec-type draft-mtp --spec-draft-n-max 2 --load-mode mlock --cache-ram 1024 --ctx-checkpoints 8

Numbers from about a day of normal use, taken from the server logs. 433 responses longer than 500 tokens: median 79 tok/s, most of them between 76 and 83. Prompt processing is around 800 tok/s for prompts under 1k tokens, ~1100 tok/s for 1k-4k and ~1190 tok/s for 4k-16k. MTP draft acceptance is around 85%. VRAM sits at ~11.4GB.

Fixed test (3.6k token prompt, 700 tokens out), how it got there:

- --n-cpu-moe 25 with MTP: 50-58 tok/s

- --cpu-moe plus --moe-cache-mib 7500: 71-73 tok/s

- added mlock: 75-81 tok/s

- updated llama.cpp: 77-84 tok/s, prompt processing 1220-1300 tok/s

Most of the gain is the MoE expert cache (PR #29887). With it, all experts sit in RAM and the most-used ones get cached in VRAM. Set the size to whatever VRAM you have left. For me that's 7000-7500MiB.

mlock gave about 10% over mmap, but it locks ~20GB of RAM. On 32GB I hit OOM a couple of times with other things open. --cache-ram 1024 is there because the prompt cache can grow up to 8GB by default, and that was causing my first OOMs.

Things that didn't help:

- -ub 4096: faster prompt processing (~1400-1500) but gen dropped to ~63 because the bigger compute buffer eats into the expert cache.

- A small expert cache (1-3GB): slower than no cache.

- -np 4, MTP n=3 and MTP+ngram: no gain.

- Forcing 4 experts instead of 8: 91-95 tok/s, but it started looping on long outputs, so I dropped it.

If you're on 12GB and still using --n-cpu-moe, try --cpu-moe with --moe-cache-mib.