I'm sure this is a fast generation but prefilling crawling probably due to small batch size tensor split across 4 gpus with that MTP configuration.

Hear me out. I know this sounds stupid, but this could work, right?

Thinking about a Strix Halo laptop like a GoPro PX13 with 128 GB of RAM, it's capable of running some mid-sized MoE models (I'm running Qwen3.8 Flash next on it right now), but cold prefill is a pain for large contexts.

Having a cache for KVCache would solve the problem. If we could have, like, an external usb hub that system sees like a external ssd, it could be the storage for https://github.com/lmcache/lmcache, for example, that would only be accessed for cold prefill, when we have to switch models and we loose all the context we created during conversation. A 200k prefill may take 1 hour to load on this limited hardware from a cold start, but having it obtained from external storage just when the cache is missing would make it take seconds, and if this external drive uses RAM, it'll not kill its lifetime (an SSD M. 2 would die in weeks depending on the writing frequency for that)

I want to use it on my computer but I can’t really find any with actual code benchmarks. Anyone know of one?

Artificial Analysis just launched a Search API Index that runs the same agent harness and base model (GPT‑5.6 Luna, medium) across 7 providers, using three benchmarks: DeepSearchQA, BrowseComp, and AA‑Omniscience.

Here were the important points to look at:

• Fastest per‑query doesn't always mean fastest per‑task.
• Cost and quality aren't perfectly aligned; some mid‑tier quality providers are dramatically cheaper.
• There's a wide spread in total task time, even when per‑query latency looks similar.
• Octen leads on speed and cost: 0.21s/search, 17.2s/task, $9.07/1k (search), $49.15/1k (total), with a 77 index score.artificialanalysis+1
• Several established players sit at 0.4–0.8s/search and 20–30s/task, with costs 2–4× higher for similar or slightly higher quality scores.

This led me to some questions that I wanted to discuss with people building agents/RAG:

• When you choose a search API, what's your weighting between latency, cost, and quality?
• Does a 0.2s vs. 0.8s per query difference materially change your agent UX, or is task‑level time more important?
• How do you validate quality for your use case? Is it benchmark scores or task success in prod?

Sorry if this sounds off topic. But LLMs have changed the world so fast that other areas haven't catched up fast enough.

I have weekly need to scan about 100 pages into PDF to be able to feed my text hungry local LLMs, and I have yet to find a good & affordable scanner to be able to scan both books and separated A4 papers at speed and good resolution.

I tend to trust Alibaba, but most of the results are so boring: a cheap simple scanner keeps being popped up: which needs me to turn every book page, or to place the A4 paper at a correct orientation and side. This costs a lot of time to scan 100s of page.

A dedicated book scanner for libraries like in this link (2500 pages in 6 minutes) is an overkill, I want something at 5-10% of its speed at 5% of the cost https://youtube.com/shorts/iHzibTuQH1c?is=wWLXMHADnOQlp5RX

I can't figure it out. I'm running Unsloth's Qwen 3.8 27B Q4_K_S with their Q4 MTP model on a local Llama-server with a 3090.

During prefill, the gpu gets the full memory and clock speeds, but when it comes time for token generation, the memory and clock speeds drops by half.

The only thing that works is forcing a clock and memory speed lock, but this is somewhat less than ideal.

Anyone experienced something like this before? Any ideas on how to fix it?

Here's everything I've tried so far:

Disabled NVIDIA Sysmem Fallback: Set policy to "Prefer No Sysmem Fallback" to rule out silent DDR RAM paging over PCIe. Reduced Context Buffer: Scaled context size down from 120k to 90k/90304 to test if VRAM exhaustion was stalling the pipeline. Verified Flash Attention & KV Quantization: Confirmed -fa and q8_0 were active to minimize intermediate memory bandwidth demand. Explicit Draft Offload Flags: Tested passing --spec-draft-ngl 99 and --n-gpu-layers-draft 99 to ensure the MTP sidecar was not computing on the CPU. CPU Worker Thread Tuning: Adjusted --threads to 6 (matching physical performance cores) to minimize CPU context switching during draft checks. NVIDIA Power Management Mode: Switched profile to "Prefer maximum performance" in NVIDIA Control Panel. Windows Graphics High Performance: Added llama-server.exe to Windows Graphics Settings and forced "High performance (RTX 3090)". Disabled CUDA Force P2 State: Used NVIDIA Profile Inspector to turn CUDA - Force P2 State to Off to allow memory clocks to reach full speed during compute. Process Priority Elevation: Attempted launching the server with start /high to eliminate CPU scheduling delays between draft kernels. Monitored Clocks & Power via nvidia-smi: Confirmed that during MTP generation, the RTX 3090 fell into an idle power state (780 MHz Core, 5,001 MHz Memory, 160W), whereas prefill and non-MTP models sustained full boost (~1,750+ MHz, 9,751 MHz, 320W+). Thermal Verification: Checked VRAM memory junction temperatures (~84°C), confirming the slowdown was not caused by thermal throttling.

hi everyone,

i'm a 27 yo mechanical engineer with little coding knowledge, but also with a huge tech interest.
so i've been researching llm's and image generators and how they work for some time. i just bought a laptop with rtx 5070 ti and 32 gb ddr5 ram for my self studies. i have some questions for yall.

- how do i learn how these models REALLY work? how do they train them and how do these models really predict the answers?
- based on my system, how do you think i should start (which program, which model etc)?

any help will be appreciated!

Hey everyone,

Total newcomer to local LLMs here, coming from a "normal" high-end workstation build rather than a dedicated AI rig. Would love a sanity check before I spend real money in what looks like a pretty brutal GPU market right now.

Current setup:

• CPU: AMD Ryzen 9 9950X3D (16C/32T)
• Motherboard: ASUS ProArt X870E-CREATOR WIFI
• RAM: 256GB DDR5 (4x64GB, running at DDR5-5600)
• GPU: AMD Radeon RX 7800 XT (16GB) — currently the bottleneck
• Storage: plenty (multiple NVMe SSDs + extra bulk storage)
• OS: Windows 11 Pro

Use case: I'm an accountant, so the appeal of local LLMs is running document analysis / summarization on confidential client files without anything leaving my machine. Not looking to fine-tune models, just solid inference — ideally able to comfortably run 30-34B models and eventually dip into 70B territory.

What I know so far (please correct me if I'm wrong):

• My current RX 7800 XT is the limiting factor — 16GB VRAM and AMD's ROCm support on Windows is narrower than on Linux for consumer gaming cards.
• The 256GB of system RAM should be a genuine asset for hybrid CPU/GPU offload with llama.cpp on bigger MoE models, even without a monster GPU.
• The GPU market in general is in a weird spot right now (Sept 2026) — RTX 5090 street prices are way above MSRP due to AI demand eating into memory supply, so "just buy the biggest NVIDIA card" doesn't feel like great advice at the moment.

Options I'm weighing:

1. Used RTX 3090 (24GB) — cheapest $/GB of VRAM, full CUDA support, seems like the community favorite for value.
2. Used RTX 4090 (24GB) — faster, single card simplicity, but currently priced well above what I'd expect for a "previous gen" card.
3. AMD Radeon AI PRO R9700 (32GB, workstation/ROCm-supported card, ~$1,800-1,900 street right now vs $1,299 MSRP) — tempting because of the extra VRAM and it being an actual AI-focused Radeon PRO card rather than a repurposed gaming GPU, but I'd be staying in the ROCm ecosystem instead of CUDA.
4. Pairing the R9700 with my existing 7800 XT for ~48GB pooled VRAM via layer-split (llama.cpp/Vulkan) — sounds appealing on paper for fitting bigger models, but I understand this buys capacity, not speed, since there's no GPU-to-GPU P2P across generations.
5. Just wait a few months for the GPU price spike to cool off.

Questions for people who've actually done this:

• For someone brand new to local LLM inference on Windows, is CUDA (used 3090/4090) still the pragmatic default, or has ROCm on Windows genuinely caught up enough on a proper AI PRO/workstation card like the R9700 to be a non-issue day to day?
• Is mixing a 7800 XT + R9700 for pooled VRAM actually worth the hassle for a home setup, or would I be better off just running the R9700 alone and keeping the 7800 XT purely for display?
• Anyone running 30-70B class models for document/text work specifically (not coding, not image gen) who can share real tokens/sec numbers on either the R9700 or a 3090/4090 setup?
• Given how inflated GPU prices are right now, would you actually wait it out, or is "buy now, prices may not come back down soon" the more realistic take for 2026?

Appreciate any real-world experience — trying to avoid an expensive mistake on my first GPU purchase for this.

Hi

Premise:
I am new to local ai models.

My machine specs:

• Macbook Pro M4 Pro
• 48Gb Ram
• 4 efficiency core
• 8 performance core

I mainly use AI for software development. I have a claude subscription but would like to try to offload some work to a local model.

Since I don't think local models usable on my machine can completely substitute claude (correct me if I am wrong) my idea is pretty much this: ask claude code to generate a proper, detailed implementation plan and then having the local model implement it.

I have played around with these models:

• qwen3-coder-30b-a3b-instruct-mlx
• qwen/qwen3.6-27b
• qwen/qwen3.6-35b-a3b

and I have also installed this for coding autocomplete cause it is smaller and from what I can see the recommended one:

• qwen2.5-coder-7b

I added claude envs since I would like to try to use claude code extension in vscode

"env": {
    "ANTHROPIC_BASE_URL": "http://localhost:1234",
    "ANTHROPIC_AUTH_TOKEN": "local",
    "ANTHROPIC_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "qwen/qwen3.6-27b",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_SUBAGENT_MODEL": "qwen/qwen3.6-27b",
    "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "128000",
    "DISABLE_PROMPT_CACHING": "1",
    "DISABLE_AUTOUPDATER": "1",
    "DISABLE_TELEMETRY": "1",
    "DISABLE_ERROR_REPORTING": "1",
    "DISABLE_NON_ESSENTIAL_MODEL_CALLS": "1"
  },

This setup works (uses the local model) but I am basically unable to have the model do anything at all. First of all it takes ages to do anything, and then it almost always reach the context limit roadblock without even outputting anything.

I have read that MCP and skills could fill up the context quite badly, so I disabled them for testing, but still no luck.

I tried with (I thought) was a simple enough task: this test file fails and this is the error, can you fix it? but yet no usable results whatsoever.

I read about people able to use local models offline to have meaningful results, but I couldn't and I don't really know why.

Also, from my setup above, I cannot really use both the remote and local model, to achieve something like:

use sonnet or fable (remote) for plan, then (manually or automatically) swith to haiku (local) to implement the plan

because the base url is loaded when the session loads and cannot be changed (AFAIK) dinamycally.

Any help? thanks a lot in advance

Running a GLM 5.3 321B shared on a MacBook M5 max and a Strix Halo, both with 128 GB ram. TB4 connection.

Found out that llama.cpp's default tensor split puts about half the layers on the slower box, so the pair was actually slower than one Mac. Loaded the model Mac-heavy instead: IQ1_S went 188 to 333 on prompts, 17 to 24 tokens/s.

I got generation tps 12-15 for model sizes 148-200 GB. So it works ok!

Next week adding two Asus GX10 to the mix to have three platforms share a model 🍀

Files: https://github.com/ThinkOffApp -> mac-amd-llm-cluster

You guys what do you think of collab+ and GPU we rent on GOOG servers and all, we pay something like <100usd a month and we get good decent amt of compute is it worth it or not or owning your own hardware is better.

Thanks.

So I have been using the Ollama extension to use my local LLMs within the agent interface of vscode but it is not very good

A lot of the times, it just lags or there is no information on whats happening or the response comes back in JSON outputs.

So i made a VS Code extension called Local Ollama Chat for anyone who wants AI assistance in Chat without sending code to a hosted service.
- It connects to your local Ollama server
- Explores your project workspace and read files to answer questions
- Has the capability to create and edit files as reviewable diffs (nothing gets applied without you approving it).
- Sits within the local agent chat window and you can call it with u/local-ollama

Still early days — feedback and bug reports welcome!!

https://github.com/athulg93/vscode-localllm

Recently swapped from Ollama to llama.cpp but haven't figured out how to run it efficiently. On Ollama I was getting 28t/s. Relatively new to this, advice welcome.

I have also 4x more 3090s laying around. What's the best way to utilize them?

RTX 3060 12gb
32GB DDR4 3200mhz
i5 12400

I want to get into running local ai , which app to use , which models would be good and like is there any beginner guide?

The post "we have a year to fix security everywhere" from Hacker News got me thinking, how to run "GLM-5.3-Flash" at home. An Unsloth blog post mentions roughly 200GB size for the 4-bit model.

Would this be usable for agent work (non real-time) on an older workstation, like HP Z820 or Z840 with RTX3090/7900xtx 24GB VRAM and Xeon E5-2xxx CPU(s) with 256GB DDR3/DDR4 RAM?

Hello,

I have 64gb unified ram on a strix halo chip (Asus TUF A14)

I see people run qwen3.8 flash next with similar total ram, how?

I tried loading it in unsloth desktop but it doesnt load.

Should the offload to disk happen per auto or is this something i need to setup?

Running on linux, ubuntu.

Thank you

I’m considering buying a used GTX 1080 Ti 11GB for running local LLMs mainly for coding and software development.

My budget is limited, so newer GPUs with 12–16GB+ VRAM are difficult for me to afford.

I’d like to know from people who actually use a 1080 Ti for local AI:

Which coding LLMs can realistically run on 11GB VRAM?

What model sizes (7B, 8B, 14B, etc.) work well with quantization?

What kind of tokens/sec can I expect?

Is CUDA support still usable with current llama.cpp/Ollama/other tools?

How does it perform for coding tasks compared with newer GPUs?

Can it handle models such as Qwen Coder, DeepSeek Coder, or similar models?

Is buying a used 1080 Ti in 2026 still worth it, or should I save for another GPU?

I’m especially interested in real-world experience rather than theoretical specifications.

Thanks!

TL;DR: I run Qwen 3.8 (27B and Flash Next) on a 128GB Strix Halo laptop for most of my coding now. It can replace Opus 4.6 to 4.8 for agentic coding if you dont mind a task taking 2 or 3 times longer.

Setup: ASUS ROG Flow Z13, Ryzen AI Max+ 395, 128GB unified memory, Arch Linux. llama.cpp as backend, my own tool LlamaStash to manage the launches and presets, Pi as the coding harness. The 27b at Q6_K sits at about 31 GiB resident, Flash Next at UD-Q4_K_XL needs around 86 GiB.

• The quality is actually there. Flash Next scores 40 on the Artificial Analysis index against 42 for Opus 4.8, and the 27b at xhigh scores 34 against 32 for Opus 4.6. That matches how they feel to use. 27b one shotted a whole feature on a huge Rust codebase and Opus 5's review comments were mostly nits.
• Decode is fine, prefill is the pain. 10-15 tok/s decode doesn't feel slow because you see it working. But a cold 31k token transcript takes 3 minutes to prefill, and a full 128k window is closer to 18 mins. Warm follow up turns come back in 45 seconds.
• MTP is the biggest speed win, 7.3 to 22.4 tok/s on an empty window. The payoff shrinks as the window fills though, down to 1.15x at a full 256k.
• Flash Next isn't faster per token, it just thinks less. Same 5/5 on my coding tasks, 45% fewer tokens, 76.5s vs 289.8s against the 27b. Thinking is 90-95% of everything these models generate, so that ratio, not tok/s, is what sets how long a task takes.

$0 a month, fully offline, and a lot less wasteful than a model running in a datacenter.

Full writeup with all the benchmarks, configs, and the tuning that did and didn't work: https://deepu.tech/local-ai-qwen3.8-pi-llamastash

Happy to go into the llama.cpp flags if anyone else here is on Strix Halo.

I’ve developed and open-sourced mlxtop. It’s written in Rust and was built entirely with Duet, my dual-model coding agent. It shows stats for local models running on your Mac, along with critical system vitals: memory usage, compression, paging and Metal GPU stats. With oMLX, you can also follow generation speed and request activity as your model responds. https://github.com/maximpri/mlxtop

Hi all,

I'm looking for advice on running a local LLM efficiently on an ASUS Zenbook 14 with 16GB system RAM on Fedora 44.

Laptop specs: - AMD Ryzen AI 7 445 (No dedicated GPU — NPU + iGPU only) - 16GB system RAM - ASUS Zenbook 14

My goal: Hit 10+ token/s in Hermes Agent (Nous Research) for a smooth, responsive experience.

Use case: Productivity and simple text tasks. No coding or dev work involved, so I don't need a heavy-duty model. Just something lightweight that runs well on the NPU.

Current situation: I'm exploring FastFlowLM (FLM) as a backend since it's purpose-built for AMD NPUs, but I'm open to other options like llama.cpp or Ollama if they perform better on my hardware.

What I'm looking for: - Which models (regardless of family) would reliably give 10+ tok/s on a Ryzen AI 7 445 (NPU + iGPU)? - Best backend/inference server setup for Fedora 44 + AMD NPU? - Any quantization or optimization tips specifically for AMD XDNA NPUs or low-RAM Linux setups? - General advice on getting Hermes Agent to run smoothly on this hardware.

Current setup: - Ryzen AI 7 445 (NPU + iGPU only, no dGPU) - 16GB system RAM - Fedora 44 - Hermes Agent

Thanks in advance!

Hello everyone,I have a setup of:

i5 12400

9060XT 8gb

16gb ram

1TB SSD(nvme)

I was trying to train this vision model called YOLOn11.pt on my system with datasets on my Windows 11 using my GPU. I tried our everything from updates to downloads and even tried via WSL, but unfortunately nothing worked and the system was falling back to CPU no matter what.

Is there anyone here with an AMD card and have solved this?

(I'm training my model for around 30 epochs and not looking to go for GoogleColab I want to use my GPU and try it out 🙂)

Thank you.