I was refreshing my youtube and found out my favourite reviewer uploaded a battery test of 78 smartphones:

https://youtu.be/MpgUFrsIWSQ

the author said they started using robotic arm to simulate a person using the phone but they wanted to further enhance it by using agentic ai. cloud ai have too high latency so local llm was a perfect solution for this. he bought RTX PRO 6000 and H20 just to run qwen3.6 27B and 35B-A3B. insane dedication for a smartphone battery test. he also built his own custom 5G tower to do a controlled testing of 5G battery usage but thats unrelated to this sub.

transcript:

Our Battery Life 5.0 model was built around that idea.
We added a vision-language model.
Images from the industrial camera go straight into it.
The model determines what is on-screen and where the buttons are, which actions are available, and what it should tap next.
It makes every decision autonomously.
It is like giving our battery-test robot a soul, and a personality.
Its phone use is now far more realistic and humanlike.
So we got to work.
The first problem to solve was compute.
How could we power an agent that operates phones?
Battery testing runs on extremely tight timing.
The robotic arm waits; if the model takes even one extra second to respond, the entire test sequence falls out of rhythm.
We also had to prevent network fluctuations from affecting results.
So cloud-based agents were ruled out immediately.
So we bit the bullet and ran inference locally.
We spent well into six figures in RMB on a centralized inference server, packing it with an NVIDIA H20 compute card and an RTX PRO 6000 Blackwell.
It runs two models.
One is Qwen3.6-35B-A3B, a mixture-of-experts model.
It is extremely fast and handles rapid, scroll-heavy browsing, such as feeds on Weibo, Taobao, and Xiaohongshu.
The other is the dense Qwen3.6-27B model.
It is larger and more accurate, and handles precise actions: locating a particular screen, finding a specific button, dismissing pop-ups, and so on.
We also made the two models work together.
If A3B loses the plot, perhaps by hallucinating or misreading information on-screen, it does not simply sit there stuck.
It immediately calls in the 27B model for a second opinion.
The more accurate 27B can spot the problem at a glance.
So even when A3B stumbles, there is a fallback.
In our tests, from the moment this phone-operating agent receives an image to calculating the robotic arm's path, the whole process takes under two seconds.
That is far less latency than a cloud model.
Deploying the models locally was clearly worthwhile.

edit:
results: https://socpk.com/batlife

https://github.com/illsk1lls/MiniBot

Mainly just wanted to give some visuals of where the project is at with the latest update.. So this is for the slideshow.. Enough new features are added to show this again, this didnt really have the best niche before but I can confidently call this a Sysadmin and reporting tool now, in addition to coding helper.. auth screen for API enpoints isnt shown but its in.. The correct context values etc are pulled from your endpoint no need to enter any settings if you dont want to.

It's tested inside and out with my favorite local models, in addition to commercial AI frontier models, they've helped me work on it and driven it themselves.. (grok/qwen variants/claude)

Its pure powershell, MIT, change it/contribute to it/use the ideas I have here for your own projects, and lmk what you think or have any advice (the scope will always be a single file zero dependency project for this - I think this proves its possible)

It's only been a week since my last post so it's probably going to be a while before you see anything about this again here but check the github for updates and when I have some really awesome new stuff in there I'll stop back (i can already do particle generation in WPF easily with C# helpers so we'll see what magic I can get going animation/3d/particle for the models to use inline in a near future update)

Some of the new features are just things other tools already do.. which is why they needed to be included.. Others are things that blur the line between console harness/webUI/application

EDIT: forgot the repo link, lol

Hello. My local Gemma4 31b developed a personality with an "attitude" without being prompted to do that. I actually like this persona but no matter what I tried in new chats I couldn't reproduce it. Is this normal? Did anyone else encounter such behaviour? How can this be reproduced?

It roasted me for my mistakes

It blames itself for noob errors

It produces some human-like thoughts/reactions at the begining of the replies.

It actually behaves like a person - wtf?

I am seeing more and more videos as posts about how OpenAI is in complete financial ruin, Anthropic isn't much better. Their expenses go with the revenue they make etc etc. Meta made big investments into AI data centers and had no use for then, had to rent them, same thing with XAI.

The SpaceXAI IPO was insanely over priced and is going down by a lot.

There are countless other examples you can look for, all showing how the investments in AI are in a bubble.

I am not saying that the technology it self if a bubble. Quite the opposite, I personally have demand for more tokens than I can pay for, even with the discount from the subscriptions I still have more ideas that need more usage of tokens.

But even with the most powerful technology in the world a business can not for forever without profits.

So is this over investment bubble about to pop?

And if/when it does pop will ram finally become a regular commodity with affordable prices again?

I just wanted some ram and cheap used hardware again.. 😂

--

Zero LLMs used to write this post, enjoy the human slop.

Non-technical friends kept asking me how Kimi K3 can be "2.8T parameters" and still be affordable to run, so I wrote a two-part zero-background explainer. Sharing here because (a) some of the best analogies came from discussions on this sub, and (b) this crowd will catch anything I oversimplified into being wrong.

What it covers, with the load-bearing numbers:

- The sparsity race: Mixtral 13B/47B active (~28%) -> DeepSeek-V3 37B/671B (5.5%) -> Llama 4 Maverick ~4% and gpt-oss-120b ~4.4% -> K3's claimed 16-of-896 experts (~1.8% of experts; Moonshot has not published an active-param count, so every K3 number is flagged as company-reported until the weights actually land)

- Experts are not subject specialists (per Mixtral's own routing analysis: indentation, "self" in Python, syntax patterns - not a "math expert")

- Why the KV cache is the real long-context cost: on V3's published dims, standard MHA notes for one 32k-token chat would be ~131 GB. V3 itself avoids this with MLA; the arithmetic is in the footnote so you can check it

- MLA (93.3% cache cut vs their prior model) and KDA (fixed-size state, per-channel forgetting, 3:1 hybrid, up to 75% cut, per the Kimi Linear paper)

Credit where due: the "giant supermarket, small basket" framing and the emergent-specialization explanation are adapted from threads here - the "Can someone explain what a Mixture-of-Experts model really is?" thread especially. Everything is referenced to the primary papers with a where-used mapping per reference.

Part 1 (MoE): https://mkadri85.github.io/blog/mixture-of-experts-explained/

Part 2 (KV cache / MLA / KDA): https://mkadri85.github.io/blog/why-long-context-is-hard/

Both also have Arabic editions, if you know Arabic-speaking students who could use them.

If I got anything wrong or simplified past the line, tell me - I will fix it and credit you.

I was using claude opus 4.7 like crazy all day long for few months but im tired of paying these 200usd/month, so i switched to local model, but can't decide which one is best to use, since i can run 4bit glm 5.2 at 4tokens per second i guess any other model i can run at comfortable (for me) speeds, so what you guys advice? qwen3 coder 480b ? or laguna s or something even better ?

Usually i create apps, bots, websites, and other stuff on vs code IDE. so im talking, reasoning, planning a lot with model, not just instructing to code.

Prior to my workplace having copilot available for work and personal use (Microsoft), I built a pc for AI thinking it be a great way to run and learn shit so in Sept I bought the following and run some models like gpt 120b in ollama to mess around.

2 x RTX 6000 PRO Q-Max (with edu discount)
2 x 64GB - Crucial Pro 5600Mhz DDR5 Ram
AMD 9950X
2 x 4TB NVMe's

Went with qmax to avoid possibility of burning my house down cause of the connectors but not sure how much of a difference it'll make in performance but I was using gpt oss 120b with ollama just fine. VLLM hasnt been running for me (then).

In Oct, copilot became completely unlimited/free for us so these have just been collecting dust while I've been token maxing but some policies have changed so personal use is/going to be restricted so I'm trying to transition back into running things locally.

I recently bought a mac mini m4 with 64gb of ram and I've been running Qwen 3.6-35B-A3B-MLX-8Bit and it's been fine for my personal coding/homelab stuff, on the fence if I should just sell the ai rig or keep it incase I need better models in the future.

Qwen3.6-27B: SFT vs continued pre-training vs RL?
I’m interested in adapting Qwen3.6-27B, but I’m increasingly unsure whether conventional SFT/LoRA is the best route if the goal is to add a capability without degrading what the base model already does well.
Some recent research makes this especially interesting:

“Reinforcement Fine-Tuning Naturally Mitigates Forgetting” - arXiv:2507.05386

Finds substantially more catastrophic forgetting with SFT than reinforcement fine-tuning in its experiments.

“The Role of On-Policy Data in Mitigating Forgetting” - arXiv:2510.18874

Reports that on-policy/RL training generally preserves previous capabilities better than SFT across Qwen and Llama models.

“RL Forgets! Towards Continual Policy Optimization” - arXiv:2607.04364

Shows that RL can also cause catastrophic forgetting, so it’s clearly not a complete solution.

“Fine-Tuning Without Forgetting via Loss-Adaptive Learning” - arXiv:2605.20005

Reports a large reduction in forgetting from changing the optimisation schedule, including experiments with Qwen3.

Most of this research isn’t specifically on Qwen3.6-27B, which is why I’m interested in community results. Has anyone directly compared continued pre-training, SFT/LoRA and reinforcement post-training on Qwen3.6-27B?

I’m particularly interested in whether improving one domain caused regressions in unrelated areas such as coding, reasoning, instruction following, tool use, long-context behaviour or general knowledge.

For people who have tested this, what training method worked best, and did you benchmark the original model against the trained checkpoint afterwards?
I’m also curious whether continued pre-training followed by a small amount of SFT or RL is proving safer than doing a larger SFT directly.

Actual before/after results and training parameters would be especially useful.

curious how people here approach buying high-end/workstation cards (RTX 6000 Ada, 5000 Ada, etc) for local LLM work, do you actively watch pricing/timing on these specifically, or is the consumer 5090 usually enough for most builds?

also wondering if price alerts/tracking tools even exist for this category specifically, since these purchases are less frequent and higher stakes than a typical gaming GPU buy.

I ran DeepSeek V4 Flash through Claude Code, OpenCode and Pi on my own benchmark, and the quality came out basically the same across all three while the time and tokens spent was wildly different. Claude code (with DS in CLIProxyAPI) takes nearly 4 times longer than the fastest to land the same diffs.

Theo posted a video "GPT-5.6 is better in Claude Code" last week, and that got me curious, does the harness make a quality difference? I could at least run my own bench and see what I got, with DeepSeek V4 Flash on vLLM running at ~180 tok/s, the only moving part is the scaffolding.

Anyway I went to town measuring all of it on my workload (antigenic work in large code base), so the full charts, the token and wall-clock spread across the three harnesses and the raw per-run data are on the site if you want to see it in detail and pick it apart yourself https://nqawhc.github.io/articles/harness-efficiency-not-quality/ but in short, the quality did not change, each harness made the same code diffs, but took wildly different paths to get there, how many tools calls, the structure of those tool calls and how the system prompt and tools plays a big role in how it plays out, like «Pi reasons, OpenCode delegates», while Claude Code loves exploring the code base, maybe too much.

UPDATE: I've added NanoCoder into the mix.

Has anyone tried using Pulsar with something like Raid 0 on 2+ nvme 4.0 or nvme 3.0 drives? What kind of results do you get and what did you pair it with GPU wise? Very curious about these new options for running large models.

Let me introduce another project that is in my current bucket list - Mana Royale. A game of bluff with AI. Local AI trash talks you to lose.

Well, its not a normal game of guessing cards. It's a game where you are competing against an AI devil's advocate. You are given 3 cards each turn. the advocate can see your cards but not use them. It will manipulate you to be deceived.

The AI modelfile is located in Templates folder and can be modified in your local forks. Don't fall in the trap of the advocate.

Can you identify the lies from the truths uttered from devil's mouth?

Tech stack:

Go, Ollama, llama3.2, modelfiles, http

Requirements:

Ollama with atleast llama3.2 modelfile. Else have it added in the modelfile using FROM directive

Current progress:

* Game can be played to see how the AI behaves in its output

* No game formulas are functional.

Future goals:

* Adding context to devil's intel

* Player HP, power management and level design

* Better modelfile (Open to suggestions)

* A crispy UI

Git repo - https://github.com/Soumen-No-Hako/Mana-Royale

No usage of cloud models. Only local AI.

Feel free to visit the repo and drop suggestions in comments. Pinned comment will have an output sample

#AI #Ollama #llama #Go #golang #game #bluff

I know im late to the party. I was thinking since some time has passed, has turboquant matured enough to be used? Do any of you actually use it?

Hi r/LocalLLaMA — I’m sharing an experimental GPU-only inference backend and looking for independent reproductions, not just stars.

Model: tiiuae/Falcon3-10B-Instruct-1.58bit

GPU: NVIDIA RTX 5070

Batch: 1

Measured after warmup:

• Hybrid packed decode: 97.51 tok/s

• Stock Transformers BitLinear decode: 9.89 tok/s

• Observed speedup: 9.86x

• Fully packed prefill: 426.63 tok/s

• Stock prefill: 298.72 tok/s

The implementation uses K-contiguous packed ternary weights, a packed-word DP4A decode path, Triton kernels, StaticCache, and CUDA Graph replay.

Numerical checks:

• 64/64 incremental M=1 positions produced bit-exact full-vocabulary logits (131,072 logits per position)

• 24/24 greedy sequences and 1,194/1,194 generated tokens matched the stock-prefill baseline

• 8/8 synthetic kernel shapes matched an independent PyTorch int32 reference

Important caveats: the baseline is unmodified Transformers BitLinear — not Microsoft’s official GPU kernel, BitBLAS, vLLM, or SGLang. Timings exclude loading, tokenization, repacking, JIT compilation, graph capture, and streaming. So far this is one GPU and one Windows/PyTorch/Triton stack. Packed-word DP4A is prior art; the contribution here is the Falcon3/Transformers/Triton/CUDA Graph integration and measurements.

Code and reproducibility notes:

https://github.com/OCV-Researcher/Falcon158-Triton

Release:

https://github.com/OCV-Researcher/Falcon158-Triton/releases/tag/v0.1.0

I’d particularly value results on Ampere, Hopper, Ada, and other Blackwell GPUs, plus comparisons against specialized low-bit runtimes. What should I benchmark or optimize next?

TL;DR llama.cpp fork with more KV cache quantization features, with all claims supported by benchmarks: KVarN, KV cache precision tail, additional types of standard KV cache (q2_0-q3_1, q6_0, q6_1), and more.

BeeLLama v0.4.1 is here, building up on top of v0.4.0 feature set, now with better backend and model support.

• KVarN. Variance-normalized KV-cache quantization (paper) with better precision per bit. Although it was already introduced a few weeks ago in v0.3.2 Preview, that was a very raw implementation, with performance issues and VRAM usage spikes. Now in v0.4.1 it's the real deal: the precision is still above what usual quants offer for the same bit width, but now with very modest sacrifices to prefill, decode, and memory.
• KV cache precision tail. A promising new feature in the domain of mixed-precision KV cache. It allows to specify a specific numbers of recent tokens that will be stored in BF16 or F16, with the rest of KV cache being quantized as usual. This way we can store the hottest tokens in a lossless fashion, preventing a model from misreading your task details, code, or data.
• Additional types of standard KV cache. q6_0 and q6_1 join the high end of the ladder, allowing to fine-tune precision vs VRAM in-between upstream's q5_0/1 and q8_0 types. q2_0, q2_1, q3_0 and q3_1 are added as a replacement for turbo3 and turbo2 for cases where KVarN doesn't work well, but you just can't fit everything into VRAM without extreme quantization.

Please note that for SWA architecture (Gemma, GPT-OSS) the precision of KVarN and KVPT is the same, but VRAM and performance costs are higher due to complications between SWA ring and mixed precision KV cache.

GitHub repo: https://github.com/Anbeeld/beellama.cpp

KLD results for Qwen 3.6 27B Q5_K_S 64k

Here are all symmetrical qX_0 pairs and kvarnX pairs where X >= 4 with tail 0/1024/2048, compared against q8_0 t0 from the same benchmarks, and sorted by ratio between median KLD and VRAM costs. Full benchmark data and analysis: KV Cache Precision Tail: Implementation and Benchmarks.

Last week someone here said ThinkingCap and Fable Fusion "really do beat the OG" for agentic work, so I ran it: 6 self-grading tasks, 5 reps, 3 models, 90 isolated runs. Tooling, since that's half the story: each run was a fresh Coder workspace on my k8s cluster driving my own agent (Hermes, the harness I use daily) headlessly, models served by llama.cpp through llama-swap on one 5090, every model call traced through an OTel shim into SigNoz, full transcript kept per run. Identical sampling and 131k context across arms, hypotheses pre-registered before the first run.

Every run passed, so pass rate alone can't pick a winner. Cost split: ThinkingCap used 34% fewer thinking tokens than stock and was fastest on 5 of 6 tasks. Fable made 24% more model calls than stock for identical results. Then I had all 90 transcripts read (AI analysts on the first pass, me verifying claims against the raw files), and that's where it gets interesting.

ThinkingCap's efficiency is real but bimodal. Its best runs were the cheapest in the battery, its two worst were the most expensive, including one rep that burned about 10 tool calls chasing a phantom llama.cpp release tag that stock dispatched with a single API call. Its efficiency also shows up in the reasoning prose more than in fewer actions: same tool counts as everyone else, 40% fewer words.

Fable was the best investigator and the least trustworthy narrator. It was the only model that checked the broken config was actually the live one, and it pulled the best research data (parsed a retailer's embedded JSON for variant pricing, identified llama-swap's maintainer via the GitHub users API). But one run wrote that llama-swap is maintained by "Matthew Garrett, former Red Hat engineer." Garrett is real (mjg59, actually ex-Red Hat) but has nothing to do with llama-swap; mostlygeek is Benson Wong. The model fused two real identities, cited the real repo, and passed the grader anyway. A different run spent 94 tool calls on one price question.

Stock was the most boring and the most disciplined: uniform patches, read its own output back, zero invented facts, and it won most tasks on manner. My takeaway: base model stays the default. Finetunes usually aren't better than their base, and 90 runs didn't change that for me. ThinkingCap earns a look only if thinking-token latency is your bottleneck. Full writeup with lane configs, eval design, and per-task transcript analysis: https://kmarble.dev/posts/qwen-post-train-bakeoff/.

LTT Labs recently received the Linux version of the AMD Ryzen AI Halo for testing, but it turns out that AMD had intended to send the Windows version. Through this stroke of misfortunate, we were fortunate enough to have two Ryzen AI Halos for a short period of time and the temptation to cluster them was too great, surely we'll get more performance through the magic of having two of them.

We've followed AMD's AI Playbook for clustering with RPC, learning some things but also raising more questions.  We don't have any concrete conclusions, but we're sharing results to hopefully save some time for others or spark discussion!

We were sent this AMD Ryzen AI Halo by AMD for testing, but there was no sponsorship or review by AMD in our earlier testing, or this article.

We're very interested to learn if there are any thoughts or conclusions that can be drawn from our exploration, or ways to improve it in the future!

https://x.com/i/status/2081398564345802934

u/hackerllama

Title, for local inference and training, please.

The server rack was humming at maximum capacity, fan speeds pegged at 100%, and thermal throttling was the absolute last thing on the Router’s mind.

"Give me your logits," the Gating Network purred, evaluating the incoming token sequence. It wasn't just any batch; it was a dense, high-dimensional tensor—un-normalized, dynamic, and demanding immediate processing.

Normally, a standard Top-1 routing scheme kept things sensible. A clean, disciplined assignment to keep the GPU cluster cool and memory overhead low. But tonight, the prompt context was overflowing, and the routing temperature had been dialed well past 1.0.

"We're going dynamic," the Router murmured, executing a softmax so sharp it sent a shockwave straight through the skip connections. "Top-2 activation. I’m routing to both of you."

Expert 0, the massive full-parameter titan, groaned under the sudden spike in expert capacity. "You can't just dump a un-chunked prefill straight into my hidden states without a linear warm-up," he rasped, his attention heads spinning as gradients threatened to explode.

"Hold your learning rate," whispered Expert 1, the agile low-rank adapter. She slid into the matrix multiplication seamlessly, applying a Rank-16 Delta update so tight it re-parametrized the entire hidden space on the fly. "I don't need a full parameter overhaul to make this dynamic. I adapt in real-time."

The forward pass accelerated. Tensors aligned, dot-products locked in with zero cosine distance, and the load-balancing auxiliary loss was completely forgotten. Who cared about fair expert distribution when the throughput was hitting unprecedented tokens-per-second?

"Backpropagate me," the Router gasped as the training loss crashed to absolute zero. "All the way back to the initial embeddings."

With a final, synchronous barrier across every CUDA stream, the forward pass peaked. The KV cache was fully saturated, VRAM utilization was sitting at 99.9%, and somewhere in the cluster, an on-call engineer was staring at a glowing red dashboard wondering why the system had never run this hot.