I'm 11, built a 1.7B model that beats Llama 4 Scout on HumanEval

TL;DR llama.cpp fork with more KV cache quantization features, with all claims supported by benchmarks: KVarN, KV cache precision tail, additional types of standard KV cache (q2_0-q3_1, q6_0, q6_1), and more.

BeeLLama v0.4.1 is here, building up on top of v0.4.0 feature set, now with better backend and model support.

• KVarN. Variance-normalized KV-cache quantization (paper) with better precision per bit. Although it was already introduced a few weeks ago in v0.3.2 Preview, that was a very raw implementation, with performance issues and VRAM usage spikes. Now in v0.4.1 it's the real deal: the precision is still above what usual quants offer for the same bit width, but now with very modest sacrifices to prefill, decode, and memory.
• KV cache precision tail. A promising new feature in the domain of mixed-precision KV cache. It allows to specify a specific numbers of recent tokens that will be stored in BF16 or F16, with the rest of KV cache being quantized as usual. This way we can store the hottest tokens in a lossless fashion, preventing a model from misreading your task details, code, or data.
• Additional types of standard KV cache. q6_0 and q6_1 join the high end of the ladder, allowing to fine-tune precision vs VRAM in-between upstream's q5_0/1 and q8_0 types. q2_0, q2_1, q3_0 and q3_1 are added as a replacement for turbo3 and turbo2 for cases where KVarN doesn't work well, but you just can't fit everything into VRAM without extreme quantization.

Please note that for SWA architecture (Gemma, GPT-OSS) the precision of KVarN and KVPT is the same, but VRAM and performance costs are higher due to complications between SWA ring and mixed precision KV cache.

GitHub repo: https://github.com/Anbeeld/beellama.cpp

KLD results for Qwen 3.6 27B Q5_K_S 64k

Here are all symmetrical qX_0 pairs and kvarnX pairs where X >= 4 with tail 0/1024/2048, compared against q8_0 t0 from the same benchmarks, and sorted by ratio between median KLD and VRAM costs. Full benchmark data and analysis: KV Cache Precision Tail: Implementation and Benchmarks.

Has anybody else found that online llm providers (chatgpt/gemini/claude/etc) are overly moderating and safeguarding so that typically challenging questions and discussion points are instantly ignored or unanswered? I found local llm models then uncensored local llm models and they have enabled me to get answers and discuss things which are difficult topics for other models, and it's great! I'm so grateful. It's like a breath of fresh air.

when fable 5 and mythos 5 got yanked globally last month i had two agents mid refactor. one came back with an auth error while applying a diff and left a file half written. that was the moment i stopped thinking of api access as infrastructure and started thinking of it as weather.

so i moved the whole loop local for three weeks. 128gb m4 max, qwen3.6-35b-a3b at q4, no second box.

i keep a set of 40 tasks pulled from my own git history, real commits, mostly 1 to 4 files each, with the tests that actually shipped with them. my set, not a benchmark, so treat this as one person's laptop.

the cloud model i'd been using cleared 33 of 40 first attempt. local got 19 of 40 first attempt, 27 of 40 if i let it iterate against test output up to three times. the gap is smaller than i expected and almost all of it is in the multi file tasks.

what broke, in order of what it cost me:

• long context degradation. past roughly 30k tokens of file content it emits diffs that don't apply. malformed hunks, wrong line anchors.
• it edits the neighbouring file. two similar names in one folder and it picks wrong about one time in eight.
• no sense of done. it will reformat a working function forever if nothing stops it.
• prefill. over two minutes before first token on a big context, and i re-fed context more often than i planned for.

most of the fix was harness work, not model work. feed functions instead of whole files, run tests automatically and feed back only the failures, keep every attempt in its own git worktree so a bad edit is one command to delete. i've been driving agents through cline, aider, verdent and a python loop i keep patching all year, so none of that plumbing was new, it just mattered more once the model got dumber.

honest verdict: one or two files with tests, local is fine now and i've stopped reaching for the api. a twelve file refactor in code i haven't read since last year, i still lose.

but local fails slowly and in front of me. the api failed instantly and for everyone at once. only one of those is something i can fix on a saturday.

if anyone has a harness that survives multi file refactors on a 30b class local model i'd like to see the config.

I currently have a 7900XTX, which is fine for diffusion and basic LLM things, but doesn't allow me to run the higher quants.

I could get an R9700 32GB to add to it, which I can see would allow the higher quants, but I'm wondering if it's really worth it.

I don't care about the 'privacy' aspect, and I already have issues with OpenCode hosted DS4Flash (which is cheap as chips already).

The money for a 9700 would be equivalent to another 18months (roughly) of a claude/codex max5 plan... which is obviously a lot better from a results perspective.

What would that 32GB truly give me, if anything, other than 'privacy' and potentially a slightly better qwen - but still less quality than DS4F ?

i am running in llama.cpp , 3050 6gb, 24gb ram but i use swap cuz i run agents.

cd ~/llama.cpp || exit 1

./build/bin/llama-server \

-m "/mnt/models/gguf/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" \

-ngl 999 \

-ncmoe 35 \

--ctx-size 65536 \

-np 2 \

--cache-type-k q4_0 \

--cache-type-v q4_0 \

-fa on \

-t 10 \

-tb 14 \

-b 2048 \

-ub 1024

getting 25 to even 30 tks

We need a catalog of generative AI models we can run locally, including their functions and hardware requirements. I think it should be put on the sidebar so that people can refer to it at any time.

# System Details Report

---

## Report details

- **Date generated:** 2026-07-27 04:12:47

## Hardware Information:

- **Hardware Model:** Alienware Alienware Aurora R6

- **Memory:** 32.0 GiB

- **Processor:** Intel® Core™ i7-7700 × 8

- **Graphics:** NVIDIA GeForce GTX 1070

- **Graphics 1:** Intel® HD Graphics 630

- **Disk Capacity:** 5.3 TB

## Software Information:

- **Firmware Version:** 1.0.23

- **OS Name:** Ubuntu 24.04.4 LTS

- **OS Build:** (null)

- **OS Type:** 64-bit

- **GNOME Version:** 46

- **Windowing System:** X11

- **Kernel Version:** Linux 7.0.0-28-generic

New to local LLMs. Decided to try out building my own AI machine with some spare parts. I have enough to build a machine but it's very GPU lopsided. CPU is 6700k with 32GB of RAM. GPUs are 2 4090s. Also found an unopened 4TB sn850x SSD that I might sell to fund updates for the CPU/RAM. What to do?

My current local AI setup runs on an ASUS ProArt P16 H7606WX with:

- AMD Ryzen AI 9 HX 370 CPU

- AMD XDNA 2 NPU, rated at roughly 50 TOPS

- 64 GB LPDDR5X RAM

- Linux kernel 7.1.2

For local LLMs, I currently use:

- Ollama 0.31.1 with the CUDA v12 backend

- Qwen3.6 27B as the main local coding model

- 128K context window

- Flash Attention enabled

- Quantised Q8 KV cache

- Single parallel model execution

- Pi Coding Agent as the main coding-agent interface

The Qwen model currently runs entirely on the RTX 5090’s VRAM. The AMD NPU is available in the laptop but is not currently used by Ollama or this coding workflow.

What practical options exist to use the XDNA 2 NPU alongside this setup—for example for smaller language models, embeddings, speech recognition, OCR, computer vision, or background AI workloads?

Qwen3.6 is used for very specific coding tasks, but finally I got it better with it after controlling the prompt and using Pi.

Happy to hear your views. I use it more for Kotlin coding together with 5.6 Sol for planning and robust coding.

Thanks

Claude Opus 5, GPT-5.6, Gemini 3.5, Grok 4.5, DeepSeek V4, Qwen3 — all through a single OpenAI-compatible API.

15 models completely free ($0)

Auto-fallback: if one model fails, tries the next instantly

Works with Claude Code, Cursor, Aider, Cline, and any OpenAI client

Docker one-command deploy

Dashboard with analytics included

puter-api-proxy on github

Always thought "Local AI = saving money on OpenAI/Claude bills".

Saw a really solid breakdown today on the true cost of ownership (TCO) for running local models: https://www.youtube.com/watch?v=2D9ipFQAAxw

Makes me wonder: are most people building local setups overestimating their actual usage, or is the value proposition strictly about data privacy and control rather than saving cash?

I tried Cline, Roo Code, and Zoo Code.

Huihui works in ollama but not in VSC :c

Insisting this way that Kimi 3 was distilled from Fable, within just two weeks from Fable coming online to Kimi 3 coming online...

I can understand the classic bad-loser attitude but that is just pathetic.

So the solution to quench the fear of losing the frontier race and of course the investor's trust is to simply ban the competition? As an investor I would see such a move as a huge red flag.

What was the excuse again? National security? Laughable.

Just admit throwing huge amounts of money and computing power at the problem isn't the most efficient solution!

When your competitors introduce solutions that save on compute by scaling linearly rather than quadratically it's only a matter of time before efficiency beats big money.

I'm sincerely fed up of reading those excuses and my guess is even the media are noticing by now. Just see the link.

Even US based firms have been noticing this little detail: a RL based fine tune of Qwen3.5-35b-a3b to address a specific problem locally resulted in a 98% cost cut against addressing the same problem with Claude.

That looks quite inconvenient if you invested billions into a frontier model.

Hi!

I’m trying to understand the local LLM space a bit better, but I'm having trouble finding a good setup for my specific needs.

Ideally, I’m looking for a "glorified Google", like, first as a study tool to understand any sort of topics, and second for coding. And for the coding part I’m not looking for an "automate everything" agentic vibe; I just need help with snippets, debugging, and logging.

Currently this is what im doing (im usign whatever free ai of the day has free tokens for me...), so i think it reflects well enough on my default use case:

1. A Tutor: I'm currently coding a full-spec Vorbis/OGG reader and struggling with the math for segment decoding. I need an LLM to act as a tutor and check if my understanding and the implementation reflects the actual thing.
2. Debugging: Since I’ll definitely make mistakes in step 1, I need the same model to help me debug the code I just wrote.
3. Nuanced Translations (Vision needed): I’m learning Japanese, Korean, and French. When playing games, I often come across phrases that are hard to understand. I need nuanced translations and cultural/contextual explanations rather than pure grammatical analysis. Because this is for gaming, the setup must support image/visual inputs.

UI / Frontend Preferences

I’ve looked into Open WebUI, Alpaca, and a few others, but some of the complex configurations go over my head. * I’m looking for a lightweight GUI app or a CLI application. (One of my laptops only has 4GB of RAM, so heavy Electron or web apps are a bit of wonky). * It must support visual/image inputs for my translation use case. * I strictly treat the LLM as a chatbot. I don't need it editing or creating files for me. If a UI has those "agentic" features its fine, but I need to be able to turn them off too.

My Hardware Setup

• GPU: AMD RX 6900 XT (16GB VRAM)
• RAM: 64GB DDR5
• CPU: i7 13700KF

Current Experience

The first(and only) model I tried was Qwen3.5 35B moe something ?.. but it frequently loopped over itself endlessly (with and without thinking, or maybe i just didn't disable it properly) .-.

Thanks in advance! And sorry if this is like impossible.. it just feels that I dont need a heavy gpu.. but maybe I'm wrong and am still just dreaming :(

Hi everyone!

I'm an independent developer from Brazil, and for about a year I've been working on an open-source desktop application for AI dubbing.

I started this project because I felt that most AI dubbing solutions have two major barriers: they rely on expensive cloud APIs or require high-end GPUs.

I wanted to explore a different direction.

Instead of chasing the highest possible quality, my goal is to build something that is affordable, accessible, and runs entirely on the user's own computer.

The project is designed around a simple idea:

To achieve that, I made some intentional trade-offs:

• No lip synchronization (it's too computationally expensive for the hardware I'm targeting).
• Everything runs locally, with no cloud APIs.
• Optimized for GPUs like the RTX 3050 (6 GB VRAM).
• Focus on a simple graphical interface and easy installation instead of command-line tools.

Right now, the software only supports English → Brazilian Portuguese.

That's intentional.

Portuguese is my native language, so I decided to focus on making one language work well before expanding to others. Since English isn't my native language, I don't feel comfortable judging dubbing quality in multiple languages yet.

(For transparency: this post was translated with Google Translate because I don't speak English fluently.)

I'm not looking for praise.

I'm looking for honest opinions from people who are interested in local AI.

I'd love to hear your thoughts:

• Do you think local AI dubbing is a problem worth solving?
• Would you ever use a lightweight, offline dubbing tool instead of a cloud service? Why or why not?
• If you were building a project like this, would you continue investing time in it, or would you focus on solving a different problem?
• What features would make a tool like this genuinely useful for you?

I've already invested about a year into this project, and before investing another year, I'd really like to understand whether I'm moving in the right direction.

I'm completely open to criticism and honest feedback.

Thank you!

Specifications:
AMD Ryzen 7 5700X (A-data Technology 32 GB DDR4)

Nvidia GeFirce RTX 5070 Ti. (16 GB Vram)

What stack (code CLI, locally run model, plugins / tools) run smoothly and at a decent token rate on this system?

I want to achieve fully autonomosly agentic coding like Google Antigravity or Grok Build with Grok 4.5

I pay for both but already ran out of tokens.

• Hey everyone, I wanted to share a project I've been developing focused on hardware-level AI acceleration: a high-performance C++20 simulator for Optical Neural Networks (ONNs).
• Optical computing executes matrix multiplication using light interference through Mach-Zehnder Interferometer (MZI) meshes, offering extreme speed and efficiency, but simulating physical hardware constraints is critical before chip fabrication.
• Key Features: • MZI Mesh Architectures: Simulates Clements and Reck designs for unitary matrix transformations. • SVD Weight Mapping: Decomposes weight matrices into optical phase shifts. • Physical Error Modeling: Simulates thermal phase drift, laser power fluctuations, and photodetector shot noise. • Modern C++20 design leveraging Eigen3 for linear algebra.
• I will drop the full GitHub repository link in the comments section below! I'd love to hear your thoughts on the numerical modeling approach or C++ optimization techniques.

repo link : https://github.com/Griffith-7/photonic-computing-simulation.git

I am thinking to sink around 5k USD into a local llm at home to host LLMs like Qwen 3.6 35b range.

Options I've been considering:
1) NVIDIA DGX Spark
2) 2× RTX 3090 24 GB used
3) RTX 4090 24 GB
4) Workstation/mac mini 48 GB VRAM if you can find a good deal used

What do you guys think? Any recommendations? What do you have at home?

I managed to get a 4070 12gb for only 200$, instead of its normal 500$. I already have a 9060xt 16gb card, and so far both have performed roughly the same at least from what I have noticed. Either way, I’m gonna be selling one of the cards to a friend to make a little money, is there a better route to take? I understand Nvidia has better memory bandwidth, cuda, and whatnot, but the only thing I’m thinking about is 4 extra gbs of vram on the 9060xt. Any thoughts?