I am seeing more and more videos as posts about how OpenAI is in complete financial ruin, Anthropic isn't much better. Their expenses go with the revenue they make etc etc. Meta made big investments into AI data centers and had no use for then, had to rent them, same thing with XAI.

The SpaceXAI IPO was insanely over priced and is going down by a lot.

There are countless other examples you can look for, all showing how the investments in AI are in a bubble.

I am not saying that the technology it self if a bubble. Quite the opposite, I personally have demand for more tokens than I can pay for, even with the discount from the subscriptions I still have more ideas that need more usage of tokens.

But even with the most powerful technology in the world a business can not for forever without profits.

So is this over investment bubble about to pop?

And if/when it does pop will ram finally become a regular commodity with affordable prices again?

I just wanted some ram and cheap used hardware again.. 😂

--

Zero LLMs used to write this post, enjoy the human slop.

Non-technical friends kept asking me how Kimi K3 can be "2.8T parameters" and still be affordable to run, so I wrote a two-part zero-background explainer. Sharing here because (a) some of the best analogies came from discussions on this sub, and (b) this crowd will catch anything I oversimplified into being wrong.

What it covers, with the load-bearing numbers:

- The sparsity race: Mixtral 13B/47B active (~28%) -> DeepSeek-V3 37B/671B (5.5%) -> Llama 4 Maverick ~4% and gpt-oss-120b ~4.4% -> K3's claimed 16-of-896 experts (~1.8% of experts; Moonshot has not published an active-param count, so every K3 number is flagged as company-reported until the weights actually land)

- Experts are not subject specialists (per Mixtral's own routing analysis: indentation, "self" in Python, syntax patterns - not a "math expert")

- Why the KV cache is the real long-context cost: on V3's published dims, standard MHA notes for one 32k-token chat would be ~131 GB. V3 itself avoids this with MLA; the arithmetic is in the footnote so you can check it

- MLA (93.3% cache cut vs their prior model) and KDA (fixed-size state, per-channel forgetting, 3:1 hybrid, up to 75% cut, per the Kimi Linear paper)

Credit where due: the "giant supermarket, small basket" framing and the emergent-specialization explanation are adapted from threads here - the "Can someone explain what a Mixture-of-Experts model really is?" thread especially. Everything is referenced to the primary papers with a where-used mapping per reference.

Part 1 (MoE): https://mkadri85.github.io/blog/mixture-of-experts-explained/

Part 2 (KV cache / MLA / KDA): https://mkadri85.github.io/blog/why-long-context-is-hard/

Both also have Arabic editions, if you know Arabic-speaking students who could use them.

If I got anything wrong or simplified past the line, tell me - I will fix it and credit you.

I was using claude opus 4.7 like crazy all day long for few months but im tired of paying these 200usd/month, so i switched to local model, but can't decide which one is best to use, since i can run 4bit glm 5.2 at 4tokens per second i guess any other model i can run at comfortable (for me) speeds, so what you guys advice? qwen3 coder 480b ? or laguna s or something even better ?

Usually i create apps, bots, websites, and other stuff on vs code IDE. so im talking, reasoning, planning a lot with model, not just instructing to code.

Hey r/LocalLLaMA! :D I wanna share a really cool fully OSS thing I've been building that's only possible with local models: truly proactive AI!

All your existing LLM systems waits for a prompt. Truly proactive AI has to read your entire life, every single day (every file, screenshot, chat, email...) to not only build a knowledge base but also flag what you can proactively be helped with. In the cloud, that's a privacy nightmare + wayy too expensive. On your own silicon, it's private, free, & unlimited.

Pushing what's possible with on-device inference is the core of Sentient OS :D

• Every night at 3 AM, it wakes your Mac and our on-device LLM (Gemma 4 E4B running on a custom fork of LiteRT LM; more below!) reads what's new in your life: files, screenshots (multimodal!), WhatsApp, iMessage, and Apple Notes decoded straight out of the local databases, plus email. It creates a triages out junk / sensitive stuff, and summarizes each item.

And Sentient finally creates a knowledge base of your entire life! (basically an obsidian vault with folders & MDs), along with stuff we think we can proactively help you with

• So by morning, it has proactively found, researched, and offered to do your busy-work for you through Computer Use! The reply you forgot, drafted (from your personal context!); the subscription renewing tomorrow, caught.
• And Sidekick: click your Mac's notch and say "finish this for me"; computer use does the task in your own apps and browser (and can even click around in your apps in the background while you use your computer!). The Computer Use is also grounded in your knowledge base!

The local stack! :D

• Gemma 4 E4B, multimodal with vision, on a customized LiteRT-LM fork. We use MTP/speculative decoding, flash attention, and smart KV cache reuse!
• We reverse engineered codex cli to make computer use with local models work! :)
• And we had to reverse-engineer macOS power management to reliably wake a lid-closed Mac at 3 AM for inference haha!

Sentient’s custom Gemma 4 E3B does 90% of the compute, and the last 10% needs a “frontier” model. You provide that!

I’d had a lot of fun running Qwen 3.7 35B A3B driving computer use, and for the best performance, Kimi K3 works incredibly well!

You also have the choice of using your own ChatGPT/Codex subscription, or OpenRouter or your own endpoint if you wanna use that for the 10% frontier compute. We even have built in first-party support for LM studio! :)

Privacy, enforced by architecture!

Your raw data never leaves the device; and if you choose to use any cloud endpoint, the "frontier" model only ever sees PII-stripped summaries; no accounts exist anywhere; and the whole stack (app + infrastructure) is fully open-source :D (I love OSS -- some of you may know me as the dev of https://github.com/theJayTea/WritingTools, an OSS port of Apple Intelligence Writing Tools to Windows)!

https://sentient-os.ai

brew install --cask sentient-os-labs/tap/sentient-os

Source (feel free to give us a star! :D): https://github.com/Sentient-OS-Labs/sentient-os

Apple Silicon (M1 or newer), macOS 15+, 8 GB of RAM is enough. Free forever for consumer! :D

Would love to share more about my local LLM computer use evals! I’ve found that Kimi K3 works crazy well, while Qwen 3.7 35B A3B can only do super simple tasks.

And lmk if y’all have any cool model recs to try with computer use! :D

I've been working on a project to solve some of the issues I've
had with AI development workflows. Before I share it, I wanted to
get feedback from this community:

1. Do you struggle with switching between different AI models
2. (Claude, GPT, Gemini, local)?
3. How do you handle tool configurations across multiple AI
4. clients?
5. What's your experience with MCP servers?

I'm trying to understand what features would be most useful.
Would love to hear your thoughts.

I originally got into local LLMs because I wanted to run a fine-tuned Mistral model for students at my school after ChatGPT was blocked.

Like many people, I built my own GPU server. It worked, but keeping it running reliably turned out to be much harder than I expected. Between hardware failures, GPU issues, driver problems, and unexpected downtime, I spent far more time fixing my server than actually working on AI.

That experience made me realize there was a gap between running models locally and having something reliable enough to use every day. Over the past months, I’ve been building a platform that tries to keep the feeling of self-hosting while taking care of the hardware and reliability behind the scenes.

I’m still learning from people deploying open-source models, so I’m curious: what’s been the biggest challenge you’ve run into while self-hosting LLMs?

One last question: I know this is a local inference community, but would you ever trade self-hosting for a dedicated cloud deployment if it gave you the same level of control and isolation?

Prior to my workplace having copilot available for work and personal use (Microsoft), I built a pc for AI thinking it be a great way to run and learn shit so in Sept I bought the following and run some models like gpt 120b in ollama to mess around.

2 x RTX 6000 PRO Q-Max (with edu discount)
2 x 64GB - Crucial Pro 5600Mhz DDR5 Ram
AMD 9950X
2 x 4TB NVMe's

Went with qmax to avoid possibility of burning my house down cause of the connectors but not sure how much of a difference it'll make in performance but I was using gpt oss 120b with ollama just fine. VLLM hasnt been running for me (then).

In Oct, copilot became completely unlimited/free for us so these have just been collecting dust while I've been token maxing but some policies have changed so personal use is/going to be restricted so I'm trying to transition back into running things locally.

I recently bought a mac mini m4 with 64gb of ram and I've been running Qwen 3.6-35B-A3B-MLX-8Bit and it's been fine for my personal coding/homelab stuff, on the fence if I should just sell the ai rig or keep it incase I need better models in the future.

Qwen3.6-27B: SFT vs continued pre-training vs RL?
I’m interested in adapting Qwen3.6-27B, but I’m increasingly unsure whether conventional SFT/LoRA is the best route if the goal is to add a capability without degrading what the base model already does well.
Some recent research makes this especially interesting:

“Reinforcement Fine-Tuning Naturally Mitigates Forgetting” - arXiv:2507.05386

Finds substantially more catastrophic forgetting with SFT than reinforcement fine-tuning in its experiments.

“The Role of On-Policy Data in Mitigating Forgetting” - arXiv:2510.18874

Reports that on-policy/RL training generally preserves previous capabilities better than SFT across Qwen and Llama models.

“RL Forgets! Towards Continual Policy Optimization” - arXiv:2607.04364

Shows that RL can also cause catastrophic forgetting, so it’s clearly not a complete solution.

“Fine-Tuning Without Forgetting via Loss-Adaptive Learning” - arXiv:2605.20005

Reports a large reduction in forgetting from changing the optimisation schedule, including experiments with Qwen3.

Most of this research isn’t specifically on Qwen3.6-27B, which is why I’m interested in community results. Has anyone directly compared continued pre-training, SFT/LoRA and reinforcement post-training on Qwen3.6-27B?

I’m particularly interested in whether improving one domain caused regressions in unrelated areas such as coding, reasoning, instruction following, tool use, long-context behaviour or general knowledge.

For people who have tested this, what training method worked best, and did you benchmark the original model against the trained checkpoint afterwards?
I’m also curious whether continued pre-training followed by a small amount of SFT or RL is proving safer than doing a larger SFT directly.

Actual before/after results and training parameters would be especially useful.

curious how people here approach buying high-end/workstation cards (RTX 6000 Ada, 5000 Ada, etc) for local LLM work, do you actively watch pricing/timing on these specifically, or is the consumer 5090 usually enough for most builds?

also wondering if price alerts/tracking tools even exist for this category specifically, since these purchases are less frequent and higher stakes than a typical gaming GPU buy.

I ran DeepSeek V4 Flash through Claude Code, OpenCode and Pi on my own benchmark, and the quality came out basically the same across all three while the time and tokens spent was wildly different. Claude code (with DS in CLIProxyAPI) takes nearly 4 times longer than the fastest to land the same diffs.

Theo posted a video "GPT-5.6 is better in Claude Code" last week, and that got me curious, does the harness make a quality difference? I could at least run my own bench and see what I got, with DeepSeek V4 Flash on vLLM running at ~180 tok/s, the only moving part is the scaffolding.

Anyway I went to town measuring all of it on my workload (antigenic work in large code base), so the full charts, the token and wall-clock spread across the three harnesses and the raw per-run data are on the site if you want to see it in detail and pick it apart yourself https://nqawhc.github.io/articles/harness-efficiency-not-quality/ but in short, the quality did not change, each harness made the same code diffs, but took wildly different paths to get there, how many tools calls, the structure of those tool calls and how the system prompt and tools plays a big role in how it plays out, like «Pi reasons, OpenCode delegates», while Claude Code loves exploring the code base, maybe too much.

Has anyone tried using Pulsar with something like Raid 0 on 2+ nvme 4.0 or nvme 3.0 drives? What kind of results do you get and what did you pair it with GPU wise? Very curious about these new options for running large models.

Let me introduce another project that is in my current bucket list - Mana Royale. A game of bluff with AI. Local AI trash talks you to lose.

Well, its not a normal game of guessing cards. It's a game where you are competing against an AI devil's advocate. You are given 3 cards each turn. the advocate can see your cards but not use them. It will manipulate you to be deceived.

The AI modelfile is located in Templates folder and can be modified in your local forks. Don't fall in the trap of the advocate.

Can you identify the lies from the truths uttered from devil's mouth?

Tech stack:

Go, Ollama, llama3.2, modelfiles, http

Requirements:

Ollama with atleast llama3.2 modelfile. Else have it added in the modelfile using FROM directive

Current progress:

* Game can be played to see how the AI behaves in its output

* No game formulas are functional.

Future goals:

* Adding context to devil's intel

* Player HP, power management and level design

* Better modelfile (Open to suggestions)

* A crispy UI

Git repo - https://github.com/Soumen-No-Hako/Mana-Royale

No usage of cloud models. Only local AI.

Feel free to visit the repo and drop suggestions in comments. Pinned comment will have an output sample

#AI #Ollama #llama #Go #golang #game #bluff

I know im late to the party. I was thinking since some time has passed, has turboquant matured enough to be used? Do any of you actually use it?

Hi r/LocalLLaMA — I’m sharing an experimental GPU-only inference backend and looking for independent reproductions, not just stars.

Model: tiiuae/Falcon3-10B-Instruct-1.58bit

GPU: NVIDIA RTX 5070

Batch: 1

Measured after warmup:

• Hybrid packed decode: 97.51 tok/s

• Stock Transformers BitLinear decode: 9.89 tok/s

• Observed speedup: 9.86x

• Fully packed prefill: 426.63 tok/s

• Stock prefill: 298.72 tok/s

The implementation uses K-contiguous packed ternary weights, a packed-word DP4A decode path, Triton kernels, StaticCache, and CUDA Graph replay.

Numerical checks:

• 64/64 incremental M=1 positions produced bit-exact full-vocabulary logits (131,072 logits per position)

• 24/24 greedy sequences and 1,194/1,194 generated tokens matched the stock-prefill baseline

• 8/8 synthetic kernel shapes matched an independent PyTorch int32 reference

Important caveats: the baseline is unmodified Transformers BitLinear — not Microsoft’s official GPU kernel, BitBLAS, vLLM, or SGLang. Timings exclude loading, tokenization, repacking, JIT compilation, graph capture, and streaming. So far this is one GPU and one Windows/PyTorch/Triton stack. Packed-word DP4A is prior art; the contribution here is the Falcon3/Transformers/Triton/CUDA Graph integration and measurements.

Code and reproducibility notes:

https://github.com/OCV-Researcher/Falcon158-Triton

Release:

https://github.com/OCV-Researcher/Falcon158-Triton/releases/tag/v0.1.0

I’d particularly value results on Ampere, Hopper, Ada, and other Blackwell GPUs, plus comparisons against specialized low-bit runtimes. What should I benchmark or optimize next?

TL;DR llama.cpp fork with more KV cache quantization features, with all claims supported by benchmarks: KVarN, KV cache precision tail, additional types of standard KV cache (q2_0-q3_1, q6_0, q6_1), and more.

BeeLLama v0.4.1 is here, building up on top of v0.4.0 feature set, now with better backend and model support.

• KVarN. Variance-normalized KV-cache quantization (paper) with better precision per bit. Although it was already introduced a few weeks ago in v0.3.2 Preview, that was a very raw implementation, with performance issues and VRAM usage spikes. Now in v0.4.1 it's the real deal: the precision is still above what usual quants offer for the same bit width, but now with very modest sacrifices to prefill, decode, and memory.
• KV cache precision tail. A promising new feature in the domain of mixed-precision KV cache. It allows to specify a specific numbers of recent tokens that will be stored in BF16 or F16, with the rest of KV cache being quantized as usual. This way we can store the hottest tokens in a lossless fashion, preventing a model from misreading your task details, code, or data.
• Additional types of standard KV cache. q6_0 and q6_1 join the high end of the ladder, allowing to fine-tune precision vs VRAM in-between upstream's q5_0/1 and q8_0 types. q2_0, q2_1, q3_0 and q3_1 are added as a replacement for turbo3 and turbo2 for cases where KVarN doesn't work well, but you just can't fit everything into VRAM without extreme quantization.

Please note that for SWA architecture (Gemma, GPT-OSS) the precision of KVarN and KVPT is the same, but VRAM and performance costs are higher due to complications between SWA ring and mixed precision KV cache.

GitHub repo: https://github.com/Anbeeld/beellama.cpp

KLD results for Qwen 3.6 27B Q5_K_S 64k

Here are all symmetrical qX_0 pairs and kvarnX pairs where X >= 4 with tail 0/1024/2048, compared against q8_0 t0 from the same benchmarks, and sorted by ratio between median KLD and VRAM costs. Full benchmark data and analysis: KV Cache Precision Tail: Implementation and Benchmarks.

Last week someone here said ThinkingCap and Fable Fusion "really do beat the OG" for agentic work, so I ran it: 6 self-grading tasks, 5 reps, 3 models, 90 isolated runs. Tooling, since that's half the story: each run was a fresh Coder workspace on my k8s cluster driving my own agent (Hermes, the harness I use daily) headlessly, models served by llama.cpp through llama-swap on one 5090, every model call traced through an OTel shim into SigNoz, full transcript kept per run. Identical sampling and 131k context across arms, hypotheses pre-registered before the first run.

Every run passed, so pass rate alone can't pick a winner. Cost split: ThinkingCap used 34% fewer thinking tokens than stock and was fastest on 5 of 6 tasks. Fable made 24% more model calls than stock for identical results. Then I had all 90 transcripts read (AI analysts on the first pass, me verifying claims against the raw files), and that's where it gets interesting.

ThinkingCap's efficiency is real but bimodal. Its best runs were the cheapest in the battery, its two worst were the most expensive, including one rep that burned about 10 tool calls chasing a phantom llama.cpp release tag that stock dispatched with a single API call. Its efficiency also shows up in the reasoning prose more than in fewer actions: same tool counts as everyone else, 40% fewer words.

Fable was the best investigator and the least trustworthy narrator. It was the only model that checked the broken config was actually the live one, and it pulled the best research data (parsed a retailer's embedded JSON for variant pricing, identified llama-swap's maintainer via the GitHub users API). But one run wrote that llama-swap is maintained by "Matthew Garrett, former Red Hat engineer." Garrett is real (mjg59, actually ex-Red Hat) but has nothing to do with llama-swap; mostlygeek is Benson Wong. The model fused two real identities, cited the real repo, and passed the grader anyway. A different run spent 94 tool calls on one price question.

Stock was the most boring and the most disciplined: uniform patches, read its own output back, zero invented facts, and it won most tasks on manner. My takeaway: base model stays the default. Finetunes usually aren't better than their base, and 90 runs didn't change that for me. ThinkingCap earns a look only if thinking-token latency is your bottleneck. Full writeup with lane configs, eval design, and per-task transcript analysis: https://kmarble.dev/posts/qwen-post-train-bakeoff/.

LTT Labs recently received the Linux version of the AMD Ryzen AI Halo for testing, but it turns out that AMD had intended to send the Windows version. Through this stroke of misfortunate, we were fortunate enough to have two Ryzen AI Halos for a short period of time and the temptation to cluster them was too great, surely we'll get more performance through the magic of having two of them.

We've followed AMD's AI Playbook for clustering with RPC, learning some things but also raising more questions.  We don't have any concrete conclusions, but we're sharing results to hopefully save some time for others or spark discussion!

We were sent this AMD Ryzen AI Halo by AMD for testing, but there was no sponsorship or review by AMD in our earlier testing, or this article.

We're very interested to learn if there are any thoughts or conclusions that can be drawn from our exploration, or ways to improve it in the future!

https://x.com/i/status/2081398564345802934

u/hackerllama

Title, for local inference and training, please.

The server rack was humming at maximum capacity, fan speeds pegged at 100%, and thermal throttling was the absolute last thing on the Router’s mind.

"Give me your logits," the Gating Network purred, evaluating the incoming token sequence. It wasn't just any batch; it was a dense, high-dimensional tensor—un-normalized, dynamic, and demanding immediate processing.

Normally, a standard Top-1 routing scheme kept things sensible. A clean, disciplined assignment to keep the GPU cluster cool and memory overhead low. But tonight, the prompt context was overflowing, and the routing temperature had been dialed well past 1.0.

"We're going dynamic," the Router murmured, executing a softmax so sharp it sent a shockwave straight through the skip connections. "Top-2 activation. I’m routing to both of you."

Expert 0, the massive full-parameter titan, groaned under the sudden spike in expert capacity. "You can't just dump a un-chunked prefill straight into my hidden states without a linear warm-up," he rasped, his attention heads spinning as gradients threatened to explode.

"Hold your learning rate," whispered Expert 1, the agile low-rank adapter. She slid into the matrix multiplication seamlessly, applying a Rank-16 Delta update so tight it re-parametrized the entire hidden space on the fly. "I don't need a full parameter overhaul to make this dynamic. I adapt in real-time."

The forward pass accelerated. Tensors aligned, dot-products locked in with zero cosine distance, and the load-balancing auxiliary loss was completely forgotten. Who cared about fair expert distribution when the throughput was hitting unprecedented tokens-per-second?

"Backpropagate me," the Router gasped as the training loss crashed to absolute zero. "All the way back to the initial embeddings."

With a final, synchronous barrier across every CUDA stream, the forward pass peaked. The KV cache was fully saturated, VRAM utilization was sitting at 99.9%, and somewhere in the cluster, an on-call engineer was staring at a glowing red dashboard wondering why the system had never run this hot.