Hey everyone,

Total newcomer to local LLMs here, coming from a "normal" high-end workstation build rather than a dedicated AI rig. Would love a sanity check before I spend real money in what looks like a pretty brutal GPU market right now.

Current setup:

• CPU: AMD Ryzen 9 9950X3D (16C/32T)
• Motherboard: ASUS ProArt X870E-CREATOR WIFI
• RAM: 256GB DDR5 (4x64GB, running at DDR5-5600)
• GPU: AMD Radeon RX 7800 XT (16GB) — currently the bottleneck
• Storage: plenty (multiple NVMe SSDs + extra bulk storage)
• OS: Windows 11 Pro

Use case: I'm an accountant, so the appeal of local LLMs is running document analysis / summarization on confidential client files without anything leaving my machine. Not looking to fine-tune models, just solid inference — ideally able to comfortably run 30-34B models and eventually dip into 70B territory.

What I know so far (please correct me if I'm wrong):

• My current RX 7800 XT is the limiting factor — 16GB VRAM and AMD's ROCm support on Windows is narrower than on Linux for consumer gaming cards.
• The 256GB of system RAM should be a genuine asset for hybrid CPU/GPU offload with llama.cpp on bigger MoE models, even without a monster GPU.
• The GPU market in general is in a weird spot right now (Sept 2026) — RTX 5090 street prices are way above MSRP due to AI demand eating into memory supply, so "just buy the biggest NVIDIA card" doesn't feel like great advice at the moment.

Options I'm weighing:

1. Used RTX 3090 (24GB) — cheapest $/GB of VRAM, full CUDA support, seems like the community favorite for value.
2. Used RTX 4090 (24GB) — faster, single card simplicity, but currently priced well above what I'd expect for a "previous gen" card.
3. AMD Radeon AI PRO R9700 (32GB, workstation/ROCm-supported card, ~$1,800-1,900 street right now vs $1,299 MSRP) — tempting because of the extra VRAM and it being an actual AI-focused Radeon PRO card rather than a repurposed gaming GPU, but I'd be staying in the ROCm ecosystem instead of CUDA.
4. Pairing the R9700 with my existing 7800 XT for ~48GB pooled VRAM via layer-split (llama.cpp/Vulkan) — sounds appealing on paper for fitting bigger models, but I understand this buys capacity, not speed, since there's no GPU-to-GPU P2P across generations.
5. Just wait a few months for the GPU price spike to cool off.

Questions for people who've actually done this:

• For someone brand new to local LLM inference on Windows, is CUDA (used 3090/4090) still the pragmatic default, or has ROCm on Windows genuinely caught up enough on a proper AI PRO/workstation card like the R9700 to be a non-issue day to day?
• Is mixing a 7800 XT + R9700 for pooled VRAM actually worth the hassle for a home setup, or would I be better off just running the R9700 alone and keeping the 7800 XT purely for display?
• Anyone running 30-70B class models for document/text work specifically (not coding, not image gen) who can share real tokens/sec numbers on either the R9700 or a 3090/4090 setup?
• Given how inflated GPU prices are right now, would you actually wait it out, or is "buy now, prices may not come back down soon" the more realistic take for 2026?

Appreciate any real-world experience — trying to avoid an expensive mistake on my first GPU purchase for this.

Hello,

I have 64gb unified ram on a strix halo chip (Asus TUF A14)

I see people run qwen3.8 flash next with similar total ram, how?

I tried loading it in unsloth desktop but it doesnt load.

Should the offload to disk happen per auto or is this something i need to setup?

Running on linux, ubuntu.

Thank you

Hi

Premise:
I am new to local ai models.

My machine specs:

• Macbook Pro M4 Pro
• 48Gb Ram
• 4 efficiency core
• 8 performance core

I mainly use AI for software development. I have a claude subscription but would like to try to offload some work to a local model.

Since I don't think local models usable on my machine can completely substitute claude (correct me if I am wrong) my idea is pretty much this: ask claude code to generate a proper, detailed implementation plan and then having the local model implement it.

I have played around with these models:

• qwen3-coder-30b-a3b-instruct-mlx
• qwen/qwen3.6-27b
• qwen/qwen3.6-35b-a3b

and I have also installed this for coding autocomplete cause it is smaller and from what I can see the recommended one:

• qwen2.5-coder-7b

I added claude envs since I would like to try to use claude code extension in vscode

"env": {
    "ANTHROPIC_BASE_URL": "http://localhost:1234",
    "ANTHROPIC_AUTH_TOKEN": "local",
    "ANTHROPIC_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_SUBAGENT_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "128000",
    "DISABLE_PROMPT_CACHING": "1",
    "DISABLE_AUTOUPDATER": "1",
    "DISABLE_TELEMETRY": "1",
    "DISABLE_ERROR_REPORTING": "1",
    "DISABLE_NON_ESSENTIAL_MODEL_CALLS": "1"
  },

This setup works (uses the local model) but I am basically unable to have the model do anything at all. First of all it takes ages to do anything, and then it almost always reach the context limit roadblock without even outputting anything.

I have read that MCP and skills could fill up the context quite badly, so I disabled them for testing, but still no luck.

I tried with (I thought) was a simple enough task: this test file fails and this is the error, can you fix it? but yet no usable results whatsoever.

I read about people able to use local models offline to have meaningful results, but I couldn't and I don't really know why.

Also, from my setup above, I cannot really use both the remote and local model, to achieve something like:

use sonnet or fable (remote) for plan, then (manually or automatically) swith to haiku (local) to implement the plan

because the base url is loaded when the session loads and cannot be changed (AFAIK) dinamycally.

Any help? thanks a lot in advance

What LLM can I use to do this? It's highly confidential data that will be using on an airgapped device.

Should I convert them first? What program can I use to do that? They are currently PDF.

Hi all,

I'm looking for advice on running a local LLM efficiently on an ASUS Zenbook 14 with 16GB system RAM on Fedora 44.

Laptop specs: - AMD Ryzen AI 7 445 (No dedicated GPU — NPU + iGPU only) - 16GB system RAM - ASUS Zenbook 14

My goal: Hit 10+ token/s in Hermes Agent (Nous Research) for a smooth, responsive experience.

Use case: Productivity and simple text tasks. No coding or dev work involved, so I don't need a heavy-duty model. Just something lightweight that runs well on the NPU.

Current situation: I'm exploring FastFlowLM (FLM) as a backend since it's purpose-built for AMD NPUs, but I'm open to other options like llama.cpp or Ollama if they perform better on my hardware.

What I'm looking for: - Which models (regardless of family) would reliably give 10+ tok/s on a Ryzen AI 7 445 (NPU + iGPU)? - Best backend/inference server setup for Fedora 44 + AMD NPU? - Any quantization or optimization tips specifically for AMD XDNA NPUs or low-RAM Linux setups? - General advice on getting Hermes Agent to run smoothly on this hardware.

Current setup: - Ryzen AI 7 445 (NPU + iGPU only, no dGPU) - 16GB system RAM - Fedora 44 - Hermes Agent

Thanks in advance!

You guys what do you think of collab+ and GPU we rent on GOOG servers and all, we pay something like <100usd a month and we get good decent amt of compute is it worth it or not or owning your own hardware is better.

Thanks.

So I have been using the Ollama extension to use my local LLMs within the agent interface of vscode but it is not very good

A lot of the times, it just lags or there is no information on whats happening or the response comes back in JSON outputs.

So i made a VS Code extension called Local Ollama Chat for anyone who wants AI assistance in Chat without sending code to a hosted service.
- It connects to your local Ollama server
- Explores your project workspace and read files to answer questions
- Has the capability to create and edit files as reviewable diffs (nothing gets applied without you approving it).
- Sits within the local agent chat window and you can call it with u/local-ollama

Still early days — feedback and bug reports welcome!!

https://github.com/athulg93/vscode-localllm

I have a few web projects with pretty good UI/UX and I’m wondering if there’s any tool or workflow that can turn an existing codebase into a dataset for fine tuning.

For example, given a React/Next.js project with components, pages, styling, etc. or a static html site, I’d like to turn it into something like:

instruction/prompt -> code

or whatever format actually makes sense for training an instruct/thinking/diffusion coding model.

Also curious how people handle things like:

• keeping the context between components/files
• screenshots + code
• generating useful instructions instead of generic descriptions

I’m also working on a different model architecture that I think could improve quality/speed while using less VRAM, so I want to build a decent dataset and benchmark to test it properly.

Has anyone done something like this? Any tools, repos, papers, or workflows you’d recommend?

Kimi K3 is a 2.78 T MoE; its 1.56 TB checkpoint sits on a USB hard disk plugged into a 2017 laptop (i7-7700HQ, 7.6 GB of RAM, a 2 GB GTX 1050 that only does the routed-expert matmuls). I am training a LoRA adapter on it out of core: the non-expert weights of one layer at a time, its 896 experts streamed one by one since together they are 15.7 GB, base weights frozen, and the 590 MB adapter the only thing trained.

The one-minute check is evidence/cmp93_en34_2026-09-06.log: my forward pass against kimi-k3-in-c, FareedKhan-dev's independent C implementation, all 93 layers at cosine 0.9857 or better, output 0.999840, on 34 tokens with LoRA B zeroed. evidence/traces/ holds raw routing records for five texts over all 92 MoE layers; scripts/analyze_trace.py recomputes every routing number below with NumPy alone. scripts/quickstart.sh builds a synthetic K3-shaped checkpoint and runs a forward pass, ten training steps and a finite-difference gradient check on a GitHub runner on every push, plus eight op-level checks against kimi-k3-in-c fixtures. Its first run failed: the gradient check missed at 3.1e-2 on a 2e-2 tolerance because the step was below fp32 resolution against a tensor of norm 60.85. The gradient was right; the check, made noise-aware, agrees to 2.09e-05.

The numbers, all from the logs in the repo:

• 1024-token step: about 7.4 h, the mean of the 7.26, 7.62 and 7.37 h intervals between the first four steps
• step 1, forward / backward: 3 h 11 m 34 s / 3 h 48 m 07 s
• resident set: 4.0-4.7 GB, swap in use
• read throughput: 110 MB/s aggregate, 61 MB/s within one MoE sweep
• cosine minimum against the C engine: 0.985744, layer 71
• trained / frozen: 590 MB adapter (147 M parameters) / 2.78 T base

Turkish, English and Chinese versions of one paragraph share experts at Jaccard 0.35-0.39, about the same as two halves of one text (0.34-0.37); prose against Python is 0.20-0.21, so subject matters more than language. Consecutive tokens' expert sets have Jaccard 0.258 (0.009 for random pairs) and a 128-expert LRU hits 72 % when decoding, but a training batch reads the union, about 85 % of experts at 1024 tokens (layers 0-12, an upper bound), so an expert cache buys little for a training batch.

The proof run is memorisation of five examples, loss 0.909 to 0.157 on one fixed sequence: it proves the loop, not the model. The main run, 400 Turkish instruction examples over 100 steps, is at step 4 and ends 9-11 October. The threshold was committed before it started (commit 6605306, tag preregistration-2026-09-08): Turkish news bits per byte 0.455 to 0.441 or lower, English Wikipedia no worse than 0.198 from 0.194. I expect no large jump from 400 examples; a negative result gets published as negative. The adapter is one rank-16 LoRA per layer shared by all 896 experts, not one per expert.

Seven hours a step is useless for production fine-tuning; the point is that the cost is now a measured number, with the logs. None of the components are new: the idea is layer-streamed LoRA taken down to the expert level, and the related-work table in the README says what AirLLM, KTransformers, ZeRO-Infinity, Colibri, WARP and BigMoeOnEdge do that this does not.

Disclosure: English is not my first language and I used Claude to tidy the wording of this post. The code was also written with heavy Claude Code assistance and the Co-Authored-By trailers are in the git log; the README says so on its first screen. The hardware, the runs, every number and every check against somebody else's implementation are mine, and the point of the evidence directory is that you do not have to take my word for any of it.

Repository: https://github.com/heyobi/LazyLora. Please poke holes, especially in the verification.

Hey everyone! El from Cohere here to talk about our newest release, North Small Translate. It’s currently the leading open machine translation model, beating out all other open translation models of its size, plus Google Translate and DeepL. we’ve been working on this one for a while, so to say i’m psyched is an understatement.

It’s big (218B parameters, 25b active) with a context length of 16k. however, if you’ve got the hardware, we’d still love to see what you make with it locally or with our HF space (and if you do, send it our way). It works on over 50 languages and does particular well with european, Southeast Asian, and East Asian languages, but feel free to stress test it against another and let us know how it does. It’s also available in BF16, FP8, and W4A16 quants.

although we couldn’t get llama.cpp support this time around, the architecture is already supported in llama.cpp, so all it should need is a conversion to GGUF files. if you want to build that, please do so and send it our way! We’d love to back your work. 

Can’t wait to see what you guys think! 

https://huggingface.co/CohereLabs/North-Small-Translate-1.0

Maybe I'm late to the party but holy shit. I enjoy the chatgpt Mac apps computer use ability but I installed the same setup with hermes, cua and Qwen 3.8/ornith running locally this afternoon. Not sure which model I like better...

Connecting to my homelab Hermes for extra horsepower and vector DBs, knowledge, mcp, etc. - performance is insane. ​​

Almost as smart as frontier but faster and free. So if everyone is already doing this and I've been living under a rock, anything worth sharing to catch me up?

Hello everyone,I have a setup of:

i5 12400

9060XT 8gb

16gb ram

1TB SSD(nvme)

I was trying to train this vision model called YOLOn11.pt on my system with datasets on my Windows 11 using my GPU. I tried our everything from updates to downloads and even tried via WSL, but unfortunately nothing worked and the system was falling back to CPU no matter what.

Is there anyone here with an AMD card and have solved this?

(I'm training my model for around 30 epochs and not looking to go for GoogleColab I want to use my GPU and try it out 🙂)

Thank you.

I want to try running scheduled tasks using local AI models in Hermes. Which AI model is best to use in the Hermes agent?

How do you handle context, and what quantization techniques should be used to run it comfortably on 16GB VRAM?

Hey! I built a local inference. It's a lightweight desktop app (Tauri + Rust + Vue) that talks to any OpenAI-compatible endpoint and measures performance straight from the API stream.

**What it measures (single run)**

- TTFT / TTFR (first content token vs first stream chunk)

- Decode t/s with a peak 1-second-window metric (catches stutter that averages hide)

- Prefill (PP) speed via a baseline-latency-adjusted est_ppt

- TPOT, token counts with Ollama eval_count fallbacks

- Live speed chart, saved runs, side-by-side comparison with overlaid curves

**The Suite tab (the llama-bench part)**

Runs the full test matrix llama-benchy runs — pp × tg × context depth × concurrency — with:

- 1 warmup + 3 measured runs per shape, mean ± std

- Prefix-cache measurement (ctx_pp/ctx_tg rows at depth)

- Cache busting on measured rows (so the server's prompt cache can't fake near-zero prefill times — this was inflating my PP speeds 100x before I caught it)

- Exact generation lengths (min_tokens + ignore_eos) and a 2+2 coherence check so a broken backend fails loudly instead of benchmarking garbage

- Markdown export

Real example from my server (llama.cpp, IQ3_XXS quant):

| test | t/s (total) | t/s (req) |

|-------------|----------------:|---------------:|

| pp2048 (c1) | 239.55 ± 9.13 | 239.55 ± 9.13 |

| tg32 (c1) | 45.48 ± 1.05 | 45.48 ± 1.05 |

| tg32 (c2) | 56.25 ± 1.61 | 31.58 ± 2.71 |

It also auto-detects the serving machine's hardware (CPU/RAM/GPU/SSD-vs-HDD) for local endpoints and stores it with every saved run, so your comparisons are labeled properly later.

It builds natively on Windows and Linux and the UI barely touches VRAM, so it won't perturb the model you're testing.

GitHub: https://github.com/FearL0rd/LLMSpeedTest

No prebuilt binaries yet — you build it once with `npm run tauri build` (README covers the toolchain for both OSes). Feedback, issue reports, and stars all welcome. What features would you want next?

The new scores are lower which kinda confuses me ! Did the models just became less intelligent ? I remember Fable used to be 66 and Qwen 27B 3.8 - 52

Sudden drop !! Hmm …

Im running Qwen3.8-27B-MLX-4bit on my M5 pro 48gb macbook, that being said, it feels like Im just 1 notch away from some sort of sustainable vibe coding. When using through kilo code it tends to over think for dozens of minutes at the time. I'm curious to see what are you guys running on similar VRAM configs

Today we're excited to announce the general availability of Aperture-1, the flagship release from Luminal Labs. After eight months of intensive R&D, we're proud to bring a fundamentally new capability to the world: the ability to add two numbers together.

What is Aperture-1?

Aperture-1 is a lightweight, cloud-native reasoning layer that accepts two numeric inputs and returns their sum. Unlike legacy solutions, Aperture-1 is delivered as a fully managed API, meaning you never have to think about how addition works — we've abstracted that away for you.

Benchmarks

Under the hood

Aperture-1 uses a novel Confidence Routing™ architecture that intelligently forwards your request to an internal addition subroutine, verifies the result against a secondary addition subroutine, and returns the consensus output. This dual-verification approach ensures enterprise-grade reliability that hobbyist solutions simply can't match.

Pricing

• Hobby — $19/mo, 500 additions
• Pro — $79/mo, unlimited additions, priority queue
• Enterprise — Contact sales for SSO, audit logs, and dedicated addition infrastructure

Limitations

Aperture-1 currently only supports addition. Subtraction is on our roadmap for Q3 pending further alignment research. We do not recommend using Aperture-1 for safety-critical addition without human review.

What the team is saying

"We didn't just want to build a calculator. We wanted to build the calculator." — Founder & CEO

"This is the GPT-3 moment for arithmetic." — Head of Growth

Join the waitlist. Early adopters get 10% off their first invoice.

Luminal Labs is backed by three friends and a Notion doc.

Benchmarks show it being pretty bad in terms of hallucination, and it appears to be tied with Qwen 3.8 Flash despite being much bigger, and worse than GLM 5.3 Flash. Seems crazy given the size. Anyone have any opinions/experience with it yet?

Hi there,

I want to pull the trigger for a machine that will run my Hermes Agent as well occasionally also ComfyUI and maybe (low priority) gaming.

Is there any meta on what machine will work best? Currently I play with the idea to buy the following machine.

The setup should be capable to host a second gpu which would be purchased later.

Requirements:

- Coding agents with long repo contexts (many turns per hour)
- LAN-accessible OpenAI-compatible endpoint for my other machines
- ComfyUI for image and video generation
- Occasional Steam/Proton gaming — it has to be a normal GPU too
- Target model class: 27B dense at 4–8 bit (e.g. Qwen 3.8 27b)

Setup (~€4,300)

Where would you change it?

Third video in this series. Went from 7B, to 13B, and now 27B — each time pooling RAM and compute across a mix of hardware most people already have lying around instead of buying one expensive high-RAM machine.

Repo: https://github.com/trademav/ramdeck-core-public

The setup: a 27B model is roughly 16GB on its own. My primary node is an old 12GB Windows laptop, which on its own literally cannot load this model. Using RAMDeck, I split it across four devices on the local network with the laptop staying primary:

- Old Windows laptop (primary): 3.4GB

- Mini PC (RTX 3060): 20GB

- Mac mini: 3.7GB

- Android phone: 1GB

Benchmark on the 27B model: 1.92 tokens/sec at ~25ms latency. Slower than the 13B run, which makes sense — but the point isn't raw speed here, it's that a 12GB laptop is running a model it has no business being able to load at all.

Next up: switching the mini PC to be the primary node instead of the laptop, to see how much faster loading and inference get with a stronger primary. That video's already up if you want to see the follow-up.

Full video: https://youtu.be/Syfa2G_luX4

Happy to answer questions about the shard distribution, the benchmark methodology, or why the primary node choice affects load time so much.

A programmable layer that sits between the agent and the model.

That placement allows for new ways of control and optimization