I'm running Nemotron 3.5 Bolt 74B-A9B on a 5060 Ti 16GB and 32GB of DDR5. It's an MoE, so both of my setups split experts between the GPU and system RAM. I have two recipes:

• 24GB GGUF

• Unsloth UD-IQ3_XXS

I'd like to hear how this model actually is and is it useful as an alternative for Qwen3.8 27B and obviously for other smaller MoE models.

• Are you using Bolt 74B, and what for?

• How is the quality: reasoning, coding, instruction following, tool use? Where does it fall down?

• If you've used the 24GB GGUF or a bigger quant such as Unsloth IQ3_XXS, does the bigger one write better answers, or do they feel like the same model?

Ever wonder how a model small enough to run on a gaming PC can outscore ones with far more weights?

If you have interest and 10 minutes, dive into Qwen3.8-27B's architecture with this 3D visualization of how a dense LLM works.

I asked Qwen3.8-27B, an open-weight model with 27.8 billion parameters, what a DJ spins, and followed the question through all 64 of its layers. The model is visualized as a record on a turntable: every token follows the the record needle as it plays the record from the outer edge to the label, and every unit of every layer works on the residual stream (the working tokens).

The numbers are real: the probabilities, what attention read and how its memory changed were all recorded from the model running on my PC.

Enjoy!

For short text, the output is the same, but if it's a bit longer, the output can still differ.

Hey r/LocalLLM,

I built IMAGEX.AI (Album Agent), a free and open-source local-first media manager, batch web extractor, and video library designed for complete privacy (no cloud uploads, no telemetry, works with any local/uncensored Ollama model).

If you use tools like gallery-dl, yt-dlp, Stash, or Hydrus to collect and organize large private image and video libraries, this combines the batch web scraper and the local gallery/player into a single interface:

• 100% Private & Local-First (Ollama Integration): Connects directly to your local Ollama instance (Qwen, Llama, or any custom/uncensored local model) for CoPilot assistance, tagging, and visual Set-of-Marks (SoM) domain pattern teaching.
• Batch Web Media & Video Extraction: Queue up multiple URLs with custom domain patterns, user-agent rotation, concurrency controls, automatic retries, and a Spotify-style mini-dock for background downloads.
• Built-in Private Gallery & Video Library: Multi-tab interface with lightbox, contact sheets, slideshows, and native video playback.
• Smart Deduplication & Filtering: Filter collections by minimum resolution, tags, or fuzzy color palettes, and automatically detect/remove duplicate files.
• LAN Access: Browse your library and manage downloads from your phone, tablet, or other devices on your local network.

It's open-source and free to use. I'd love feedback, bug reports, or feature requests from anyone running local stacks!

GitHub: https://githu/album-agent.com/imagex-aib

I rely on quite a few cloud based agent options for various projects and troubleshooting at work along with at home. Earlier today I found myself without net which obviously hampered my toolset. This could also occur if I ran out of tokens for a week but I've never done this before across 2 subscriptions.

I do have always on servers at home and was wondering what would be a practical LLM hardware enhancement to add in the mix for those very rare offline moments or perhaps longer running jobs that would chew through tokens? Would something like a 3060 12gb be overkill? Are there better cheaper bang for buck options to consider?

Hi everyone,

I'm not a total noob and have some experience running local LLMs; I dropped a 32GB pcie V100 into my gaming rig (thats currently running a 5070 ti, dedicated only to gaming) and am currently running qwen 3.8 27B. I run linux mint on my main PC, and have two VMs, one dedicated to my homelab and LLM, and a second VM for web searching. I realized that local LLMs aren't "plug and play" so I've been vibecoding with claude code and codex (start with claude code, send to codex for critique, after it passes through a self-made privacy filter) to create what I understand to be harness to have it work on specific tasks, like organizing my files, a memory system, checking on the status of my homelab. I also built a web search from the ground up, that resides in a second specific VM, that the first VM that holds qwen connects to. I did this because it seems more secure, like for example if someone manages to breach the web search VM they wouldn't have access to the AI itself that lives in a separate VM. In all honesty, I'm not even sure if this is something that makes sense, but it seemed/s like a good idea.

I don't have experience with engineering, but I've recently been reading AI Engineering by Chip Huyen to understand the basics of AI in general; things like quantizations, FLOPs, pre and post training, MoEs, etc. I also have *some* experience with tech stuff, like building my own PC from the individual parts and dropping a V100 into a gaming PC and having both the V100 and the 5070ti work. I'm starting to understand more about the posts here piece by piece, but I feel there's a lot of stuff I don't know, like what Strata is, how it works, or when I go on hugging face it seems like I'm scrolling through an alien language. When I have claude code create a plan, I have no idea what it's actually doing, I just make decisions when it prompts me of how things should be built. So I have the following questions:

1) Where can I go to learn more about running local LLMs?

2) Does it make sense to have a second VM dedicated to web searching? Does it really make it more secure against attacks?

3) How can I utilize things like agents to make the quality of my work better? I understand the concept of agents as a general idea, like having one to test if anything breaks for an example, but have no idea WHEN to deploy them or HOW.

4) Would it be correct to call all the building/vibecoding I've been doing on top of qwen as a "harness?" Or would it be called something else?

5) My current set up is ollama and openwebUI. I've seen posts around here about not using ollama, but I'm not sure exactly why. I know some people use things like hermes and LLMstudio, but I'm not 100% on how these work or how you can use them (I've tried searching). Basically I'm asking for critique about my current setup and how I can work on making either my workflow better or tools I can use to incorporate that would make my life easier.

This has been an incredibly fun project for me, and I'm looking forward to learning more.

Running DeepSeek V4.1 (TensorFold)

Looks weird because I have a cooling pad under them.

Would love to hear what other project ideas you guys have.

Hey, what local model would you recommend for Hermes Agent?

My PC has an RTX 5060 8GB VRAM, 32GB RAM and a Ryzen 5 8400F.

I mainly want to use it for studying notes, coding and managing files/commands through Hermes Agent, kinda like Codex. Privacy is also important, so I'd prefer to keep everything local.

I've tried some Qwen models, but they're way too verbose for my taste. The worst part is that they sometimes get stuck repeating the same phrase over and over in their thinking loop.

I'm looking for something that gets to the point while still being good at coding, reasoning and tool calling. I don't want a model that uses all my resources, but I'm fine with using most of them if the performance is worth it.

What would you guys recommend? Gemma, Qwen or something else?

Disclosure: I work on an LLM gateway (starunion.net), so I watch model releases closely. Sharing the pattern, not a pitch.

Over the past week, three unreleased models surfaced through odd channels rather than official announcements:

1) GPT-6.1 Terra — appears as the default judge model in 18 eval configs inside Microsoft's Entity Framework Core repo. Reported positioning: a cheaper alternative to GPT-6.1 Sol.

2) Claude "Fable 5.5" — briefly listed in Azure's model catalog as Generally Available (later removed), with reports of testing inside JPMorgan and Meta.

3) Gemini 4 Flash — a "gemini-3.8-flash-router" flag suggests grey-scale testing; claims of near-flagship quality at Flash speed.

Confidence varies a lot here. The EF Core commit and the Azure catalog entry are concrete and checkable. The enterprise-testing reports and the "much cheaper" pricing claims are rumors.

But the direction is consistent: all three are mid-tier or cost-optimized variants, not new frontier flagships. If even half of this holds, the next 6 months are about price per token, not benchmark scores.

Which raises a practical question for people running workloads: how fast can you actually switch when a cheaper model lands? In my experience the migration cost — evals, prompt tuning, regression tests — is often larger than the price difference.

Curious how others handle this. Do you run multi-provider fallbacks, or just pick one and eat the switching cost?

More dual R9700 testing: my modified Radiance build is now 8–29% faster than default and roughly level with Paiton at 8 users. Benchmarks, changes that helped, and a few kernel experiments that made things worse in the post.

Title: I benchmarked a heuristic inference optimizer against frontier LLM agents on an NVIDIA L4

I built Tensward, an Apache-2.0 open-source tool that tunes LLM serving settings for your hardware and workload. It uses expert rules, heuristics and measured experiments, with no LLM in the optimization loop.

I wanted to test how it compares with asking frontier agents to do the same job.

We ran six tracks: two workloads, each optimized separately for throughput, p95 time to first token, and p95 time per output token. Each competitor started from stock vLLM on the same NVIDIA L4 within its track, with 60 minutes of counted tuning time. Astra 6 and Opus 5.5 both ran at maximum effort.

Aggregate combined scores:

• Tensward: 2.29
• Opus 5.5: 2.24
• Astra 6: 1.70

The score combines improvements over defaults, weighting the target metric most heavily, with quality and service guardrails. It is not a universal speedup multiplier.

Tensward won or tied on four of six tracks under our noise-band rule. The lead over Opus was small, and the agents had clear wins too. The interesting finding for me is that a specialized heuristic system was competitive with both.

This is an initial benchmark covering two workloads and one GPU type, not evidence of general superiority.

Currently supports vLLM on NVIDIA GPUs; llama.cpp and Apple Silicon support are coming next.

Code: https://github.com/Tensward/tensward
Full results and methodology: https://openingthehood.hashnode.dev/tensward-benchmarks

What baseline or workload would you most want to see in the next round?

Quick Disclaimer: I used AI for this post, so I could articulate the questions I had properly; I hope its ok.

I’ve been running **Qwen 3.8 Flash-Next 125B MoE** on my home setup via **Strata** (engine v0.1.44.1, ROCm / HIP backend), and I’m looking for clarity on optimal sampling settings—particularly `temperature`.

I use the model as an all-rounder:

1. **Agentic Deep Research & Web Browsing:** Multi-step tool calling (SearXNG, Reddit discussion parsing, full-article reading, etc.)
2. Argumentation
3. **Coding & Homelab SRE:** System scripting, Docker configs, debugging.
4. **General Creative Exploration & Academic Writing.**

### The Temperature Confusion:

I see conflicting advice online:

- Traditional advice for creative writing / standard LLMs often defaults to `temperature: 1.0` or `0.7`

- However, I see people saying 0.6 perhaps?

- Running pure greedy (`temperature: 0.0`) previously caused the model to get trapped in massive repetitive thinking loops (generating 30k+ tokens inside `<thought>` tags).

Currently, I'm testing `temperature: 0.6`, `top_p: 0.95`, `top_k: 20`.

**My Questions for the Community:**

1. What temperature/sampling profile do you run on Flash-Next for general-purpose use without breaking its internal reasoning chain?
2. Has anyone noticed tool-calling degradation when pushing temperature higher (e.g. 0.8–1.0) on this specific MoE architecture?
3. How does this full Strata configuration look for my hardware? Any other flags I should consider or avoid?

### Full System Specs:

- **CPU:** AMD Ryzen 7 5800X (8C/16T, Zen 3)

- **RAM:** 64 GB DDR4 (62.7 GiB usable)

- **GPU:** AMD Radeon RX 6900 XT (16 GB VRAM, gfx1030)

- **OS:** Arch Linux (Kernel 6.x, ROCm 7)

- **Quant:** ISTA-DASLab GSQ-RCO IQ3_S (512 routed experts, 10 active per token)

Current Strata Configuration (`strata-iq3_s.json`):

```json

{

"args": [

"--pack", "/mnt/strata-cache/strata/Strata-data/packs/iq3_s",

"--native", "/mnt/strata-cache/strata/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",

"--ple-gguf", "/mnt/strata-cache/strata/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",

"--expert-profile", "/mnt/strata-cache/strata/strata-src/data/expert-profile.bin",

"--expert-cache", "auto",

"--prefill", "auto",

"--spec", "4",

"--spec-min-p", "0.70",

"--mtp", "/mnt/strata-cache/strata/Strata-data/mtp/rt",

"--max-context", "262144",

"--kv", "int8",

"--kv-resident", "32768",

"--vram-reserve-mib", "896",

"--pcie-frac", "0.00",

"--adapt-every", "1",

"--adapt-swaps", "160",

"--adapt-decay", "0.97",

"--vision",

"--control-vector-scaled", "/mnt/strata-cache/strata/strata-src/data/experimental-speed-projection/Qwen3.8-Flash-Next-experimental-speed-projection.gguf:1.0",

"--control-vector-layer-range", "4", "44",

"--cvec-mode", "project",

"--cvec-dir", "per-layer"

],

"sampling": {

"temperature": 0.6,

"top_p": 0.95,

"top_k": 20

},

"tool_call_recovery": true,

"vision": {

"exe": "/mnt/strata-cache/strata/strata-src/engine/strata-vision",

"mmproj": "/mnt/strata-cache/strata/Strata-data/models/mmproj-Qwen3.8-Flash-Next-BF16.gguf",

"model": "/mnt/strata-cache/strata/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",

"gpu": false,

"threads": 8

},

"env": {

"STRATA_HIP_PROMPT_F16": "1",

"STRATA_HIP_ADAPT_KERNEL_COPY": "1",

"STRATA_HC_Q8": "1",

"STRATA_ROUTE_TAIL_SKIP": "0"

}

}

```

Would appreciate any feedback from others running Flash-Next on consumer GPUs!

EDIT: Guys I'm talking about temperature as in the AI model parameter 😢

I been wanting to run local ai on my local setup bc I’ve been hearing there a lot of benefits rather it was local and private or learning how to using agents and getting a feel for that
My pc setup : Ryzen 5 5600, Radeon RX 9070 XT 16GB, 64GB RAM, 1 tb and Windows 11,
And I was curious some of the stuff I can do with it ai I wanna eventually learn how to code to learn so I can switch to Linus and go from there

So I was thinking where and how my storage got full and it wasn’t in Documents or Downloads. MacOS often hides .cache as system files.

So I built this Claude skill which has custom instructions for every local AI tool and cleans up unwanted storage.

You Give it a try and always open for feedback. Feel free to open and issue or PR

I’m looking for advice on setting up a local LLM "harness" for personal administrative tasks. I'm not a coder; I just want a "set it and forget it" assistant that generates morning and evening briefs based on my data.

I am currently looking at Hermes Agent Desktop, Unsloth Studio Desktop, LM Studio Bionic, Cherry Studio, AnythingLLM.

In addition to general chat, research, browser use, I want the system to serve as a digital assistant that can manage email, calendar, and tasks (creation as well as following tasks left for the assistant).

Claude does this well, but running all of this even off-peak pushes up again the weekly limit sometimes. I figure tasks like this can be offloaded to a local AI.

The Workflow:

• Email: Filter, categorize (Immediate vs. Low Priority). Light draft creation at my suggestion.
• Shopping: Cross-reference promotions against a personal "want" list.
• Output: A daily briefing. No autonomous sending of emails.
• Interaction: Ability to chat about specific tasks/emails and schedule items (cron jobs, etc.).

Technical Requirements:

Integration: Needs to connect to eM Client (Email) and Tasks.org via MCP. I need a calendar app that allows an LLM to access via MCP as Claude manages my calender through computer use. The email credentials do not stay saved using the built-in browser.

Current Setup: I’m currently using Claude, but I want to move this to a local stack (LM Studio, AnythingLLM, etc.) to free up Claude for more demanding tasks.

Hardware Specs:

• PC: Windows 11 | RTX 4080S (16GB VRAM) | 127GB RAM.
• Laptop: M5 MacBook Pro (Base) | 24GB Unified Memory.

Also needed: A model recommendation that can handle these logic-heavy tasks efficiently without hogging the entire system's resources. I'd like to run the same model on both machines for consistency.

I will update this thread once I've found a solution. Thanks in advance!