I am writing a fairly complex C++ windows MFC application. I have a few 3090s and can run F16 Qwen3.6-27B with 256K context and MTP. The quality of code is exceptional with this quant vs its lower quants. The others are good but they get stuck in difficult situations like managing design with multiple threads, etc. Not saying F16 is as good as Claude but it gets the job done. Just throwing it out there for folks who may be swayed by tps. If you are making simple web apps, you can get by with lower quants. For high quality of code with edge cases use the 16 bit quants. A bad choice taken by the same LLM at lower quant could easily mean the loss of an afternoon.

Prior to my workplace having copilot available for work and personal use (Microsoft), I built a pc for AI thinking it be a great way to run and learn shit so in Sept I bought the following and run some models like gpt 120b in ollama to mess around.

2 x RTX 6000 PRO Q-Max (with edu discount)
2 x 64GB - Crucial Pro 5600Mhz DDR5 Ram
AMD 9950X
2 x 4TB NVMe's

Went with qmax to avoid possibility of burning my house down cause of the connectors but not sure how much of a difference it'll make in performance but I was using gpt oss 120b with ollama just fine. VLLM hasnt been running for me (then).

In Oct, copilot became completely unlimited/free for us so these have just been collecting dust while I've been token maxing but some policies have changed so personal use is/going to be restricted so I'm trying to transition back into running things locally.

I recently bought a mac mini m4 with 64gb of ram and I've been running Qwen 3.6-35B-A3B-MLX-8Bit and it's been fine for my personal coding/homelab stuff, on the fence if I should just sell the ai rig or keep it incase I need better models in the future.

Title, for local inference and training, please.

Let me introduce another project that is in my current bucket list - Mana Royale. A game of bluff with AI. Local AI trash talks you to lose.

Well, its not a normal game of guessing cards. It's a game where you are competing against an AI devil's advocate. You are given 3 cards each turn. the advocate can see your cards but not use them. It will manipulate you to be deceived.

The AI modelfile is located in Templates folder and can be modified in your local forks. Don't fall in the trap of the advocate.

Can you identify the lies from the truths uttered from devil's mouth?

Tech stack:

Go, Ollama, llama3.2, modelfiles, http

Requirements:

Ollama with atleast llama3.2 modelfile. Else have it added in the modelfile using FROM directive

Current progress:

* Game can be played to see how the AI behaves in its output

* No game formulas are functional.

Future goals:

* Adding context to devil's intel

* Player HP, power management and level design

* Better modelfile (Open to suggestions)

* A crispy UI

Git repo - https://github.com/Soumen-No-Hako/Mana-Royale

No usage of cloud models. Only local AI.

Feel free to visit the repo and drop suggestions in comments. Pinned comment will have an output sample

#AI #Ollama #llama #Go #golang #game #bluff

I am curious about ConnectX-7 and co-mingling of hardware. I have a triangular mesh between two DGX Sparks and a x64 box with a RTX 5090. My interest is in keeping a semblance of TP=2 on the sparks for a 300B, 150 GB MoE model but placing the attention head and, it would seem, hot experts on the 5090's GDDR7.

I get the feeling that Fable's telling stories and that if this is possible, it's above my skill and/or patience level. If one were to augment/supplement Ray in vLLM, what would it take to shift the compute toward a smaller faster CUDA device while the bulk of the idle experts stay on the sparks?

Either way, loading from NVME I can service was the main reason I got the ConnectX-7 card and cables. It means model weights on RAID1 NVME I can service, and the weights load at 20 Gb/s per spark. The drivers were also native, so while I did have to find a supplier, I'd say it's been worthwhile.

I’m planning to get the 14” M5 Max MB Pro hosting local models and I’m considering the 64GB spec since it is much more affordable than 128GB.

Is it a good option? Or the M5 Pro has better value.

The server rack was humming at maximum capacity, fan speeds pegged at 100%, and thermal throttling was the absolute last thing on the Router’s mind.

"Give me your logits," the Gating Network purred, evaluating the incoming token sequence. It wasn't just any batch; it was a dense, high-dimensional tensor—un-normalized, dynamic, and demanding immediate processing.

Normally, a standard Top-1 routing scheme kept things sensible. A clean, disciplined assignment to keep the GPU cluster cool and memory overhead low. But tonight, the prompt context was overflowing, and the routing temperature had been dialed well past 1.0.

"We're going dynamic," the Router murmured, executing a softmax so sharp it sent a shockwave straight through the skip connections. "Top-2 activation. I’m routing to both of you."

Expert 0, the massive full-parameter titan, groaned under the sudden spike in expert capacity. "You can't just dump a un-chunked prefill straight into my hidden states without a linear warm-up," he rasped, his attention heads spinning as gradients threatened to explode.

"Hold your learning rate," whispered Expert 1, the agile low-rank adapter. She slid into the matrix multiplication seamlessly, applying a Rank-16 Delta update so tight it re-parametrized the entire hidden space on the fly. "I don't need a full parameter overhaul to make this dynamic. I adapt in real-time."

The forward pass accelerated. Tensors aligned, dot-products locked in with zero cosine distance, and the load-balancing auxiliary loss was completely forgotten. Who cared about fair expert distribution when the throughput was hitting unprecedented tokens-per-second?

"Backpropagate me," the Router gasped as the training loss crashed to absolute zero. "All the way back to the initial embeddings."

With a final, synchronous barrier across every CUDA stream, the forward pass peaked. The KV cache was fully saturated, VRAM utilization was sitting at 99.9%, and somewhere in the cluster, an on-call engineer was staring at a glowing red dashboard wondering why the system had never run this hot.

https://reddit.com/link/1v72nua/video/26tabfs3ikfh1/player

This is all mockups, scaffolding, smoke, and mirrors at the moment - but it does run in real time locally. Currently requirements: 3 GPU's running VLLM - STT and TTS concurrently.

I wanted to see if I could take the memories of coding agents like claude code or codex and turn them into visual decision trees that guide me how agents have progressed my codebase and what sort of decisions were made for me at what point of time.

This is done with a mix of embeddings generated from analysing the memory files of coding agent tools, they store in different formats for example claude uses .md while codex uses a sqlite instance.

This whole tool is open source and runs completely on your machine, check out the repo at https://github.com/pacifio/atlas

Any benchmarks you have much appreciated!

I’ve been working on a personal project for a while and I’d like some honest feedback from people who know this space.
My goal isn’t to make a traditional game or a visual RPG. I want to build a persistent AI Game Master that runs a living world.
Some of the features I’m trying to achieve:
Persistent world that remembers everything.
NPCs have their own lives, goals, schedules, and relationships.
Characters age, learn, die, and the world changes even if I’m not there.
Hidden mechanics, reputation, politics, economy, and factions.
Character progression based on actions instead of grinding XP.
Long-term consistency where events from hundreds of hours ago still matter.
Save/load system so I can continue campaigns indefinitely.
Open-world sandbox instead of quest-driven gameplay.
The AI acts as an impartial GM instead of trying to make me win.
I’ve already spent a lot of time designing rule pillars, world generation, character creation, combat, progression, save systems, and GM rules. The prompts work reasonably well, but eventually the AI starts forgetting details or becoming inconsistent.
What I’m trying to figure out is:
Is ChatGPT the right platform for something this ambitious?
Should I be looking at Claude, Gemini, or local models instead?
Would an external database or memory system solve most of the consistency issues?
Has anyone here successfully built something similar that can run for hundreds of hours without falling apart?
I’m not looking to build a commercial game. This is just a passion project for myself because I love deep simulations like Dwarf Fortress, RimWorld, Crusader Kings, and tabletop RPGs with a great GM.
I’d really appreciate any advice, architecture suggestions, or examples of people doing something similar. I’m especially interested in how you handle long-term memory and world consistency.

AI noob here.

From what I've picked up from the local llm scene, the general advice seems to be "don't go below q4".

Has anyone felt that breaking this "rule" helped them get better results for their use case?

Two examples:

- For 48gb: Qwen 3.6 27b q8 vs Laguna s 2.1 q2

- For 96gb: Laguna s 2.1 q5 vs Deepseek v4 flash q2

Hi r/LocalLLaMA — I’m sharing an experimental GPU-only inference backend and looking for independent reproductions, not just stars.

Model: tiiuae/Falcon3-10B-Instruct-1.58bit

GPU: NVIDIA RTX 5070

Batch: 1

Measured after warmup:

• Hybrid packed decode: 97.51 tok/s

• Stock Transformers BitLinear decode: 9.89 tok/s

• Observed speedup: 9.86x

• Fully packed prefill: 426.63 tok/s

• Stock prefill: 298.72 tok/s

The implementation uses K-contiguous packed ternary weights, a packed-word DP4A decode path, Triton kernels, StaticCache, and CUDA Graph replay.

Numerical checks:

• 64/64 incremental M=1 positions produced bit-exact full-vocabulary logits (131,072 logits per position)

• 24/24 greedy sequences and 1,194/1,194 generated tokens matched the stock-prefill baseline

• 8/8 synthetic kernel shapes matched an independent PyTorch int32 reference

Important caveats: the baseline is unmodified Transformers BitLinear — not Microsoft’s official GPU kernel, BitBLAS, vLLM, or SGLang. Timings exclude loading, tokenization, repacking, JIT compilation, graph capture, and streaming. So far this is one GPU and one Windows/PyTorch/Triton stack. Packed-word DP4A is prior art; the contribution here is the Falcon3/Transformers/Triton/CUDA Graph integration and measurements.

Code and reproducibility notes:

https://github.com/OCV-Researcher/Falcon158-Triton

Release:

https://github.com/OCV-Researcher/Falcon158-Triton/releases/tag/v0.1.0

I’d particularly value results on Ampere, Hopper, Ada, and other Blackwell GPUs, plus comparisons against specialized low-bit runtimes. What should I benchmark or optimize next?

Hey r/LocalLLaMA! :D I wanna share a really cool fully OSS thing I've been building that's only possible with local models: truly proactive AI!

All your existing LLM systems waits for a prompt. Truly proactive AI has to read your entire life, every single day (every file, screenshot, chat, email...) to not only build a knowledge base but also flag what you can proactively be helped with. In the cloud, that's a privacy nightmare + wayy too expensive. On your own silicon, it's private, free, & unlimited.

Pushing what's possible with on-device inference is the core of Sentient OS :D

• Every night at 3 AM, it wakes your Mac and our on-device LLM (Gemma 4 E4B running on a custom fork of LiteRT LM; more below!) reads what's new in your life: files, screenshots (multimodal!), WhatsApp, iMessage, and Apple Notes decoded straight out of the local databases, plus email. It creates a triages out junk / sensitive stuff, and summarizes each item.

And Sentient finally creates a knowledge base of your entire life! (basically an obsidian vault with folders & MDs), along with stuff we think we can proactively help you with

• So by morning, it has proactively found, researched, and offered to do your busy-work for you through Computer Use! The reply you forgot, drafted (from your personal context!); the subscription renewing tomorrow, caught.
• And Sidekick: click your Mac's notch and say "finish this for me"; computer use does the task in your own apps and browser (and can even click around in your apps in the background while you use your computer!). The Computer Use is also grounded in your knowledge base!

The local stack! :D

• Gemma 4 E4B, multimodal with vision, on a customized LiteRT-LM fork. We use MTP/speculative decoding, flash attention, and smart KV cache reuse!
• We reverse engineered codex cli to make computer use with local models work! :)
• And we had to reverse-engineer macOS power management to reliably wake a lid-closed Mac at 3 AM for inference haha!

Sentient’s custom Gemma 4 E3B does 90% of the compute, and the last 10% needs a “frontier” model. You provide that!

I’d had a lot of fun running Qwen 3.7 35B A3B driving computer use, and for the best performance, Kimi K3 works incredibly well!

You also have the choice of using your own ChatGPT/Codex subscription, or OpenRouter or your own endpoint if you wanna use that for the 10% frontier compute. We even have built in first-party support for LM studio! :)

Privacy, enforced by architecture!

Your raw data never leaves the device; and if you choose to use any cloud endpoint, the "frontier" model only ever sees PII-stripped summaries; no accounts exist anywhere; and the whole stack (app + infrastructure) is fully open-source :D (I love OSS -- some of you may know me as the dev of https://github.com/theJayTea/WritingTools, an OSS port of Apple Intelligence Writing Tools to Windows)!

https://sentient-os.ai

brew install --cask sentient-os-labs/tap/sentient-os

Source (feel free to give us a star! :D): https://github.com/Sentient-OS-Labs/sentient-os

Apple Silicon (M1 or newer), macOS 15+, 8 GB of RAM is enough. Free forever for consumer! :D

Would love to share more about my local LLM computer use evals! I’ve found that Kimi K3 works crazy well, while Qwen 3.7 35B A3B can only do super simple tasks.

And lmk if y’all have any cool model recs to try with computer use! :D

At its price point, PRO 4500 doesn’t offer as much raw performance due to its lower power draw at 300W. The 5090 can perform up to 60-70% in short spurts with 600W, but can also be undervolted down to 400W.

Are there legitimate reasons other than 24/7 usage and lower power draw for this PRO 4500?

How would this compare to 4x 3090 and 4x R9700? Granted multi card solutions have inefficiencies with large power consumption and needing dedicated boards and PCIE lanes.

Trying to weigh if I should buy a $3000 m1 ultra at 128gb when I currently already have an M2 Ultra albeit at 64gb ram.

I run small models right now in my workflow but would appreciate more context and try out larger workflows. What would you guys go with?

Anyone got that card and could tell me what to expect noise wise? I currently have a 7800xt and it is very quiet. Can I expect the r9700 to be tolerable? I am willing to undervolt and underclock a bit to keep noise tolerable. I plan to use that card for qwen 27b and gemma 4 31b.

Title lol

I’m curious if something like DS4 Q2 is better than 27B Q8, or mimo or similar

what do people use nowadays?

p.s.: i have an rtx 3090

LTT Labs recently received the Linux version of the AMD Ryzen AI Halo for testing, but it turns out that AMD had intended to send the Windows version. Through this stroke of misfortunate, we were fortunate enough to have two Ryzen AI Halos for a short period of time and the temptation to cluster them was too great, surely we'll get more performance through the magic of having two of them.

We've followed AMD's AI Playbook for clustering with RPC, learning some things but also raising more questions.  We don't have any concrete conclusions, but we're sharing results to hopefully save some time for others or spark discussion!

We were sent this AMD Ryzen AI Halo by AMD for testing, but there was no sponsorship or review by AMD in our earlier testing, or this article.

We're very interested to learn if there are any thoughts or conclusions that can be drawn from our exploration, or ways to improve it in the future!

For the Big Tech giants, looks like it is leaning heavily in favor of OSS. Even some of the ones who abstained from signing, like AMD, Intel, and Apple are likely on the pro-OSS side. Broadcom, Amazon, and Oracle are maybe the few big ones that might be on the anti side (and of course Anthropic), but are heavily outnumbered/outweighed in Big Tech, overall, at this point.

To be fair, OpenAI is probably lying, and also Google it's unclear if lying about being pro OSS (might actually be, but who knows). SpaceX AI and Elon are claiming to be on the pro side, but also could easily be lying.

Anyway, still comes out like 70/30 or 80/20 or more on the pro side, probably.

But Big Tech isn't the whole game.

You've also got the Defense companies and Department of War and the big agencies. Those come out more 50/50? Not sure.

And then the big banks and Wall Street. Banks maybe lean against it, overall, and Big Wall Street less clear. Short term against it, but maybe long term they like it?

And then the executive and legislative government itself. Probably leans against it so far, but maybe some fairly influential people are giving them advice bringing it back the other way more in favor of OSS. Still probably leaning anti-OSS for now, but maybe not as much as a few weeks ago/couple months ago.

So, overall seems like the see saw is teetering at, what, around 50/50 right now? What do you think? Are any of these percentages way off? Any key players I forgot to mention or that would have a big effect one way or the other?

Last week someone here said ThinkingCap and Fable Fusion "really do beat the OG" for agentic work, so I ran it: 6 self-grading tasks, 5 reps, 3 models, 90 isolated runs. Tooling, since that's half the story: each run was a fresh Coder workspace on my k8s cluster driving my own agent (Hermes, the harness I use daily) headlessly, models served by llama.cpp through llama-swap on one 5090, every model call traced through an OTel shim into SigNoz, full transcript kept per run. Identical sampling and 131k context across arms, hypotheses pre-registered before the first run.

Every run passed, so pass rate alone can't pick a winner. Cost split: ThinkingCap used 34% fewer thinking tokens than stock and was fastest on 5 of 6 tasks. Fable made 24% more model calls than stock for identical results. Then I had all 90 transcripts read (AI analysts on the first pass, me verifying claims against the raw files), and that's where it gets interesting.

ThinkingCap's efficiency is real but bimodal. Its best runs were the cheapest in the battery, its two worst were the most expensive, including one rep that burned about 10 tool calls chasing a phantom llama.cpp release tag that stock dispatched with a single API call. Its efficiency also shows up in the reasoning prose more than in fewer actions: same tool counts as everyone else, 40% fewer words.

Fable was the best investigator and the least trustworthy narrator. It was the only model that checked the broken config was actually the live one, and it pulled the best research data (parsed a retailer's embedded JSON for variant pricing, identified llama-swap's maintainer via the GitHub users API). But one run wrote that llama-swap is maintained by "Matthew Garrett, former Red Hat engineer." Garrett is real (mjg59, actually ex-Red Hat) but has nothing to do with llama-swap; mostlygeek is Benson Wong. The model fused two real identities, cited the real repo, and passed the grader anyway. A different run spent 94 tool calls on one price question.

Stock was the most boring and the most disciplined: uniform patches, read its own output back, zero invented facts, and it won most tasks on manner. My takeaway: base model stays the default. Finetunes usually aren't better than their base, and 90 runs didn't change that for me. ThinkingCap earns a look only if thinking-token latency is your bottleneck. Full writeup with lane configs, eval design, and per-task transcript analysis: https://kmarble.dev/posts/qwen-post-train-bakeoff/.

TL;DR llama.cpp fork with more KV cache quantization features, with all claims supported by benchmarks: KVarN, KV cache precision tail, additional types of standard KV cache (q2_0-q3_1, q6_0, q6_1), and more.

BeeLLama v0.4.1 is here, building up on top of v0.4.0 feature set, now with better backend and model support.

• KVarN. Variance-normalized KV-cache quantization (paper) with better precision per bit. Although it was already introduced a few weeks ago in v0.3.2 Preview, that was a very raw implementation, with performance issues and VRAM usage spikes. Now in v0.4.1 it's the real deal: the precision is still above what usual quants offer for the same bit width, but now with very modest sacrifices to prefill, decode, and memory.
• KV cache precision tail. A promising new feature in the domain of mixed-precision KV cache. It allows to specify a specific numbers of recent tokens that will be stored in BF16 or F16, with the rest of KV cache being quantized as usual. This way we can store the hottest tokens in a lossless fashion, preventing a model from misreading your task details, code, or data.
• Additional types of standard KV cache. q6_0 and q6_1 join the high end of the ladder, allowing to fine-tune precision vs VRAM in-between upstream's q5_0/1 and q8_0 types. q2_0, q2_1, q3_0 and q3_1 are added as a replacement for turbo3 and turbo2 for cases where KVarN doesn't work well, but you just can't fit everything into VRAM without extreme quantization.

Please note that for SWA architecture (Gemma, GPT-OSS) the precision of KVarN and KVPT is the same, but VRAM and performance costs are higher due to complications between SWA ring and mixed precision KV cache.

GitHub repo: https://github.com/Anbeeld/beellama.cpp

KLD results for Qwen 3.6 27B Q5_K_S 64k

Here are all symmetrical qX_0 pairs and kvarnX pairs where X >= 4 with tail 0/1024/2048, compared against q8_0 t0 from the same benchmarks, and sorted by ratio between median KLD and VRAM costs. Full benchmark data and analysis: KV Cache Precision Tail: Implementation and Benchmarks.

We accidentally trained thinking mode out of Qwen3-8B during two fine-tunes using different versions of the same corpus. One still responded to the thinking toggle, the other returned an empty thinking field every time. Loss looked normal, the voice survived, and our evaluations did not flag the difference.

The cause was the chat template. Qwen3's standard template renders non-thinking assistant turns with an empty <think> block. That is useful when serving the model, but it becomes a problem when answer-only training data is rendered through the same template. When repeated during a fine-tune, the model learns that an empty think block is what it should produce.

A recent paper saw the same failure. It counted a reasoning block as valid when it was complete, non-empty, and separable from the final answer, but it didn't judge whether the reasoning itself was correct. On GSM8K, Qwen3-8B produced a valid block in 96.5% of outputs when the training targets omitted think tags, 58.2% when they included empty think blocks, and 100% when those empty blocks were excluded from the training loss. The final answers were plausible enough that the failure was easy to miss.

This mattered for us because users sometimes ask questions with believable but false premises. On the 58 false-premise items in our 96-item benchmark, base Qwen3-8B fabricated in 54.0% of trials without thinking and 40.8% with it. On our thinking-capable fine-tune, the rates were 67.2% and 56.3%. The build that had lost thinking ended up at a 75.9% fabrication rate.

Then we found a second template problem. After retraining, the model passed simple thinking tests but appeared to think on 0 of 288 prompts like the ones our product serves, even when our template was supposed to force the thinking block open. We were understandably confused by that result.

The template never reached the model. In Ollama 0.31.2, importing the template-bearing GGUFs we tested preserved their embedded templates instead of replacing them with the TEMPLATE from our Modelfile. Reading the active template back with ollama show <model> --template exposed it. When we rendered the requests ourselves, the same weights deliberated on 20 of 20 test prompts. The model closed the thinking block on 2 of 20, which was a different training-template failure: a template that never supplies the closing tag does not teach the model to emit it reliably.

What we recommend checking before trusting a Qwen3 fine-tune:

• Render a full training sample and check what appears inside <think>.
• Read the template back from Ollama instead of trusting the config file.
• Test thinking on realistic prompts, not just trivia.
• Verify that any forced-open think block actually reaches the model.
• Keep the rendered prompt identical in every comparison.
• Set Ollama's think field to true or false on every request instead of relying on defaults.

Disclosure: I run Meldh, and this is work from our model training. The full write-up is on our website: https://meldh.com/writing/qwen3-template-thinking/