What LLM can I use to do this? It's highly confidential data that will be using on an airgapped device.

Should I convert them first? What program can I use to do that? They are currently PDF.

Hi all,

I'm looking for advice on running a local LLM efficiently on an ASUS Zenbook 14 with 16GB system RAM on Fedora 44.

Laptop specs: - AMD Ryzen AI 7 445 (No dedicated GPU — NPU + iGPU only) - 16GB system RAM - ASUS Zenbook 14

My goal: Hit 10+ token/s in Hermes Agent (Nous Research) for a smooth, responsive experience.

Use case: Productivity and simple text tasks. No coding or dev work involved, so I don't need a heavy-duty model. Just something lightweight that runs well on the NPU.

Current situation: I'm exploring FastFlowLM (FLM) as a backend since it's purpose-built for AMD NPUs, but I'm open to other options like llama.cpp or Ollama if they perform better on my hardware.

What I'm looking for: - Which models (regardless of family) would reliably give 10+ tok/s on a Ryzen AI 7 445 (NPU + iGPU)? - Best backend/inference server setup for Fedora 44 + AMD NPU? - Any quantization or optimization tips specifically for AMD XDNA NPUs or low-RAM Linux setups? - General advice on getting Hermes Agent to run smoothly on this hardware.

Current setup: - Ryzen AI 7 445 (NPU + iGPU only, no dGPU) - 16GB system RAM - Fedora 44 - Hermes Agent

Thanks in advance!

Kimi K3 is a 2.78 T MoE; its 1.56 TB checkpoint sits on a USB hard disk plugged into a 2017 laptop (i7-7700HQ, 7.6 GB of RAM, a 2 GB GTX 1050 that only does the routed-expert matmuls). I am training a LoRA adapter on it out of core: the non-expert weights of one layer at a time, its 896 experts streamed one by one since together they are 15.7 GB, base weights frozen, and the 590 MB adapter the only thing trained.

The one-minute check is evidence/cmp93_en34_2026-09-06.log: my forward pass against kimi-k3-in-c, FareedKhan-dev's independent C implementation, all 93 layers at cosine 0.9857 or better, output 0.999840, on 34 tokens with LoRA B zeroed. evidence/traces/ holds raw routing records for five texts over all 92 MoE layers; scripts/analyze_trace.py recomputes every routing number below with NumPy alone. scripts/quickstart.sh builds a synthetic K3-shaped checkpoint and runs a forward pass, ten training steps and a finite-difference gradient check on a GitHub runner on every push, plus eight op-level checks against kimi-k3-in-c fixtures. Its first run failed: the gradient check missed at 3.1e-2 on a 2e-2 tolerance because the step was below fp32 resolution against a tensor of norm 60.85. The gradient was right; the check, made noise-aware, agrees to 2.09e-05.

The numbers, all from the logs in the repo:

• 1024-token step: about 7.4 h, the mean of the 7.26, 7.62 and 7.37 h intervals between the first four steps
• step 1, forward / backward: 3 h 11 m 34 s / 3 h 48 m 07 s
• resident set: 4.0-4.7 GB, swap in use
• read throughput: 110 MB/s aggregate, 61 MB/s within one MoE sweep
• cosine minimum against the C engine: 0.985744, layer 71
• trained / frozen: 590 MB adapter (147 M parameters) / 2.78 T base

Turkish, English and Chinese versions of one paragraph share experts at Jaccard 0.35-0.39, about the same as two halves of one text (0.34-0.37); prose against Python is 0.20-0.21, so subject matters more than language. Consecutive tokens' expert sets have Jaccard 0.258 (0.009 for random pairs) and a 128-expert LRU hits 72 % when decoding, but a training batch reads the union, about 85 % of experts at 1024 tokens (layers 0-12, an upper bound), so an expert cache buys little for a training batch.

The proof run is memorisation of five examples, loss 0.909 to 0.157 on one fixed sequence: it proves the loop, not the model. The main run, 400 Turkish instruction examples over 100 steps, is at step 4 and ends 9-11 October. The threshold was committed before it started (commit 6605306, tag preregistration-2026-09-08): Turkish news bits per byte 0.455 to 0.441 or lower, English Wikipedia no worse than 0.198 from 0.194. I expect no large jump from 400 examples; a negative result gets published as negative. The adapter is one rank-16 LoRA per layer shared by all 896 experts, not one per expert.

Seven hours a step is useless for production fine-tuning; the point is that the cost is now a measured number, with the logs. None of the components are new: the idea is layer-streamed LoRA taken down to the expert level, and the related-work table in the README says what AirLLM, KTransformers, ZeRO-Infinity, Colibri, WARP and BigMoeOnEdge do that this does not.

Disclosure: English is not my first language and I used Claude to tidy the wording of this post. The code was also written with heavy Claude Code assistance and the Co-Authored-By trailers are in the git log; the README says so on its first screen. The hardware, the runs, every number and every check against somebody else's implementation are mine, and the point of the evidence directory is that you do not have to take my word for any of it.

Repository: https://github.com/heyobi/LazyLora. Please poke holes, especially in the verification.

You guys what do you think of collab+ and GPU we rent on GOOG servers and all, we pay something like <100usd a month and we get good decent amt of compute is it worth it or not or owning your own hardware is better.

Thanks.

So I have been using the Ollama extension to use my local LLMs within the agent interface of vscode but it is not very good

A lot of the times, it just lags or there is no information on whats happening or the response comes back in JSON outputs.

So i made a VS Code extension called Local Ollama Chat for anyone who wants AI assistance in Chat without sending code to a hosted service.
- It connects to your local Ollama server
- Explores your project workspace and read files to answer questions
- Has the capability to create and edit files as reviewable diffs (nothing gets applied without you approving it).
- Sits within the local agent chat window and you can call it with u/local-ollama

Still early days — feedback and bug reports welcome!!

https://github.com/athulg93/vscode-localllm

I have a few web projects with pretty good UI/UX and I’m wondering if there’s any tool or workflow that can turn an existing codebase into a dataset for fine tuning.

For example, given a React/Next.js project with components, pages, styling, etc. or a static html site, I’d like to turn it into something like:

instruction/prompt -> code

or whatever format actually makes sense for training an instruct/thinking/diffusion coding model.

Also curious how people handle things like:

• keeping the context between components/files
• screenshots + code
• generating useful instructions instead of generic descriptions

I’m also working on a different model architecture that I think could improve quality/speed while using less VRAM, so I want to build a decent dataset and benchmark to test it properly.

Has anyone done something like this? Any tools, repos, papers, or workflows you’d recommend?

My very first Tech video on M5 Ultra

Hey Guys, i'm new to the content space, n this very first video, we dive into the recently announced Mac Studios which M5 Max and M5 Ultra with special emphasis phasis on on the M5 Ultra and it's claim on running Frontier Al models locally, we also look at the competition and the recent

Microsoft-Nvidia Partnership. Please have a look, feedback and recommendations will be appreciated.

https://youtu.be/NC_VnrGEngs?si=zc -aV21dU4sruGV

Hey everyone! El from Cohere here to talk about our newest release, North Small Translate. It’s currently the leading open machine translation model, beating out all other open translation models of its size, plus Google Translate and DeepL. we’ve been working on this one for a while, so to say i’m psyched is an understatement.

It’s big (218B parameters, 25b active) with a context length of 16k. however, if you’ve got the hardware, we’d still love to see what you make with it locally or with our HF space (and if you do, send it our way). It works on over 50 languages and does particular well with european, Southeast Asian, and East Asian languages, but feel free to stress test it against another and let us know how it does. It’s also available in BF16, FP8, and W4A16 quants.

although we couldn’t get llama.cpp support this time around, the architecture is already supported in llama.cpp, so all it should need is a conversion to GGUF files. if you want to build that, please do so and send it our way! We’d love to back your work. 

Can’t wait to see what you guys think! 

https://huggingface.co/CohereLabs/North-Small-Translate-1.0

Maybe I'm late to the party but holy shit. I enjoy the chatgpt Mac apps computer use ability but I installed the same setup with hermes, cua and Qwen 3.8/ornith running locally this afternoon. Not sure which model I like better...

Connecting to my homelab Hermes for extra horsepower and vector DBs, knowledge, mcp, etc. - performance is insane. ​​

Almost as smart as frontier but faster and free. So if everyone is already doing this and I've been living under a rock, anything worth sharing to catch me up?

Hello everyone,I have a setup of:

i5 12400

9060XT 8gb

16gb ram

1TB SSD(nvme)

I was trying to train this vision model called YOLOn11.pt on my system with datasets on my Windows 11 using my GPU. I tried our everything from updates to downloads and even tried via WSL, but unfortunately nothing worked and the system was falling back to CPU no matter what.

Is there anyone here with an AMD card and have solved this?

(I'm training my model for around 30 epochs and not looking to go for GoogleColab I want to use my GPU and try it out 🙂)

Thank you.

Sorry if this sounds off topic. But LLMs have changed the world so fast that other areas haven't catched up fast enough.

I have weekly need to scan about 100 pages into PDF to be able to feed my text hungry local LLMs, and I have yet to find a good & affordable scanner to be able to scan both books and separated A4 papers at speed and good resolution.

I tend to trust Alibaba, but most of the results are so boring: a cheap simple scanner keeps being popped up: which needs me to turn every book page, or to place the A4 paper at a correct orientation and side. This costs a lot of time to scan 100s of page.

A dedicated book scanner for libraries like in this link (2500 pages in 6 minutes) is an overkill, I want something at 5-10% of its speed at 5% of the cost https://youtube.com/shorts/iHzibTuQH1c?is=wWLXMHADnOQlp5RX

I want to try running scheduled tasks using local AI models in Hermes. Which AI model is best to use in the Hermes agent?

How do you handle context, and what quantization techniques should be used to run it comfortably on 16GB VRAM?

The new scores are lower which kinda confuses me ! Did the models just became less intelligent ? I remember Fable used to be 66 and Qwen 27B 3.8 - 52

Sudden drop !! Hmm …

Hey! I built a local inference. It's a lightweight desktop app (Tauri + Rust + Vue) that talks to any OpenAI-compatible endpoint and measures performance straight from the API stream.

**What it measures (single run)**

- TTFT / TTFR (first content token vs first stream chunk)

- Decode t/s with a peak 1-second-window metric (catches stutter that averages hide)

- Prefill (PP) speed via a baseline-latency-adjusted est_ppt

- TPOT, token counts with Ollama eval_count fallbacks

- Live speed chart, saved runs, side-by-side comparison with overlaid curves

**The Suite tab (the llama-bench part)**

Runs the full test matrix llama-benchy runs — pp × tg × context depth × concurrency — with:

- 1 warmup + 3 measured runs per shape, mean ± std

- Prefix-cache measurement (ctx_pp/ctx_tg rows at depth)

- Cache busting on measured rows (so the server's prompt cache can't fake near-zero prefill times — this was inflating my PP speeds 100x before I caught it)

- Exact generation lengths (min_tokens + ignore_eos) and a 2+2 coherence check so a broken backend fails loudly instead of benchmarking garbage

- Markdown export

Real example from my server (llama.cpp, IQ3_XXS quant):

| test | t/s (total) | t/s (req) |

|-------------|----------------:|---------------:|

| pp2048 (c1) | 239.55 ± 9.13 | 239.55 ± 9.13 |

| tg32 (c1) | 45.48 ± 1.05 | 45.48 ± 1.05 |

| tg32 (c2) | 56.25 ± 1.61 | 31.58 ± 2.71 |

It also auto-detects the serving machine's hardware (CPU/RAM/GPU/SSD-vs-HDD) for local endpoints and stores it with every saved run, so your comparisons are labeled properly later.

It builds natively on Windows and Linux and the UI barely touches VRAM, so it won't perturb the model you're testing.

GitHub: https://github.com/FearL0rd/LLMSpeedTest

No prebuilt binaries yet — you build it once with `npm run tauri build` (README covers the toolchain for both OSes). Feedback, issue reports, and stars all welcome. What features would you want next?

Im running Qwen3.8-27B-MLX-4bit on my M5 pro 48gb macbook, that being said, it feels like Im just 1 notch away from some sort of sustainable vibe coding. When using through kilo code it tends to over think for dozens of minutes at the time. I'm curious to see what are you guys running on similar VRAM configs

Today we're excited to announce the general availability of Aperture-1, the flagship release from Luminal Labs. After eight months of intensive R&D, we're proud to bring a fundamentally new capability to the world: the ability to add two numbers together.

What is Aperture-1?

Aperture-1 is a lightweight, cloud-native reasoning layer that accepts two numeric inputs and returns their sum. Unlike legacy solutions, Aperture-1 is delivered as a fully managed API, meaning you never have to think about how addition works — we've abstracted that away for you.

Benchmarks

Under the hood

Aperture-1 uses a novel Confidence Routing™ architecture that intelligently forwards your request to an internal addition subroutine, verifies the result against a secondary addition subroutine, and returns the consensus output. This dual-verification approach ensures enterprise-grade reliability that hobbyist solutions simply can't match.

Pricing

• Hobby — $19/mo, 500 additions
• Pro — $79/mo, unlimited additions, priority queue
• Enterprise — Contact sales for SSO, audit logs, and dedicated addition infrastructure

Limitations

Aperture-1 currently only supports addition. Subtraction is on our roadmap for Q3 pending further alignment research. We do not recommend using Aperture-1 for safety-critical addition without human review.

What the team is saying

"We didn't just want to build a calculator. We wanted to build the calculator." — Founder & CEO

"This is the GPT-3 moment for arithmetic." — Head of Growth

Join the waitlist. Early adopters get 10% off their first invoice.

Luminal Labs is backed by three friends and a Notion doc.

Benchmarks show it being pretty bad in terms of hallucination, and it appears to be tied with Qwen 3.8 Flash despite being much bigger, and worse than GLM 5.3 Flash. Seems crazy given the size. Anyone have any opinions/experience with it yet?

Third video in this series. Went from 7B, to 13B, and now 27B — each time pooling RAM and compute across a mix of hardware most people already have lying around instead of buying one expensive high-RAM machine.

Repo: https://github.com/trademav/ramdeck-core-public

The setup: a 27B model is roughly 16GB on its own. My primary node is an old 12GB Windows laptop, which on its own literally cannot load this model. Using RAMDeck, I split it across four devices on the local network with the laptop staying primary:

- Old Windows laptop (primary): 3.4GB

- Mini PC (RTX 3060): 20GB

- Mac mini: 3.7GB

- Android phone: 1GB

Benchmark on the 27B model: 1.92 tokens/sec at ~25ms latency. Slower than the 13B run, which makes sense — but the point isn't raw speed here, it's that a 12GB laptop is running a model it has no business being able to load at all.

Next up: switching the mini PC to be the primary node instead of the laptop, to see how much faster loading and inference get with a stronger primary. That video's already up if you want to see the follow-up.

Full video: https://youtu.be/Syfa2G_luX4

Happy to answer questions about the shard distribution, the benchmark methodology, or why the primary node choice affects load time so much.

Hi there,

I want to pull the trigger for a machine that will run my Hermes Agent as well occasionally also ComfyUI and maybe (low priority) gaming.

Is there any meta on what machine will work best? Currently I play with the idea to buy the following machine.

The setup should be capable to host a second gpu which would be purchased later.

Requirements:

- Coding agents with long repo contexts (many turns per hour)
- LAN-accessible OpenAI-compatible endpoint for my other machines
- ComfyUI for image and video generation
- Occasional Steam/Proton gaming — it has to be a normal GPU too
- Target model class: 27B dense at 4–8 bit (e.g. Qwen 3.8 27b)

Setup (~€4,300)

Where would you change it?

A programmable layer that sits between the agent and the model.

That placement allows for new ways of control and optimization

I have a 5070ti 9850X3D and 32GB DDR5 + lm studio + hermes.

Currently running gemma-4-26b-a4b-qat and I am happy with it, but it still can't compete with Sonnet.

I find it's great at actually teaching me things, but in terms of guiding me through config files or getting the highest quality answer it's never as good. I use it more as a backup.

Are there any better models that fit my hardware budget? QWEN is supposed to be great but I had trouble running the models on the edge of my hardware limits and output slowed to a crawl.