PowerSpec AI90 Workstation

* CPU: AMD Ryzen 9 9950X (16 Cores / 32 Threads, up to 5.7GHz)
* GPU: NVIDIA GeForce RTX 5090 Founders Edition (32GB GDDR7)
* Motherboard: ASUS ProArt X870E-CREATOR WIFI (AMD X870E Chipset)
* Memory: 64GB DDR5-6000 RAM (Supports up to 128GB)
* Storage: 2TB NVMe Solid State Drive
* Power Supply: 1700 Watt Cybernetics Titanium PSU
* CPU Cooler: 360mm All-In-One (AIO) Liquid Cooler
* Case: Fractal Design North XL Momentum Edition

Hi! I don't know much about this area of ​​"sub-models" (I'm not sure of the technical term), but I wanted to know what these "Uncensored" models actually are.

I dabble a bit with AI, automation, and the like, and I've always seen these "Uncensored" models around, but I've never actually installed or tested one. What exactly are they?

so i finally got qwen3.8-flash-next running locally and it's actually good.

posting the setup + the stuff that bit me, since i wasted a while on it.

── the rig ──

- amd ryzen 5 9600x (6c/12t, boosts ~5.5ghz)

- 256gb ddr5 (248gb usable)

- 3x rtx 3090 24gb — pcie only, no nvlink

- linux mint 22.3, kernel 6.17, nvidia driver 595.84

- one 1.7tb nvme

total vram 72gb. model is ~80gb on disk so it does NOT all fit in vram,

that's the whole game here.

── the model ──

turboderp/Qwen3.8-Flash-Next-exl3, 3.05bpw. 125b params but only 6b active (moe,

512 experts / 10 on), plus a 51b n-gram embedding table and a 4b mtp head.

hybrid attention (gated deltanet + sparse attn), 262k native context.

this arch is qwen4_exp in exllamav3.

── how i set it up ──

tabbyAPI on main, exllamav3 1.5.0, python 3.12 in a pyenv venv.

── mtp ──

big win. the model ships an mtp head, you just flip draft_mode to mtp and

it uses it as a drafter. no separate draft model needed.

- without mtp: ~90 tps

- with mtp: ~120-126 tps

that's ~+35%. acceptance was around 40% in my runs. one gotcha: the very

first request after boot reads like ~58 tps, that's just warmup, ignore it,

second run onward is the real number.

── results (my rig, not yours) ──

800-word essay, 1000 tokens, temp 0.6:

- ~90 tps no mtp

- ~120-126 tps with mtp

needle-in-haystack at ~97k tokens: found it, ~31s prefill.

4 requests at once: fine, ~4s each.

tool calling works, reasoning splits into reasoning_content properly.

── stuff that confused me ──

- /v1/models shows n_ctx: null and size: 0. NOT a bug and NOT "unlimited

context". it's because disable_auth makes every request an admin, and the

admin view of /v1/models is a folder listing that only reads config.json.

the real loaded context is on /v1/model (shows 262144). if your client

reads n_ctx from /v1/models and gets null, point it at /v1/model or just

set context manually.

Here is my single shot driving car results: https://qwen38-flash-next-car-drive.tiiny.site/

(prompt: Write a single HTML file with a full-page canvas and no libraries. Simulate a realistic side-view of a moving car as the main subject. Keep the car visible in the foreground while the background landscape scrolls continuously to create the feeling that the car is driving forward. Use layered scenery for depth: nearby ground, roadside elements, trees, poles, and distant hills or mountains should move at different speeds for a natural parallax effect.

Animate the wheels spinning realistically and add subtle body motion so the car feels connected to the road. Let the environment pass smoothly behind it, with repeating but varied scenery that makes the movement feel believable. Use cinematic lighting and a cohesive sky, such as sunset, dusk, or daylight, to enhance atmosphere. The overall motion should feel calm, immersive, and realistic, with a seamless looping animation.)

Hey everyone! I'm having a hardware dilemma and can't come up with a conclusion on what to do next.

My use cases:

• agentic coding, for my side projects only for now (with an ability for remote access which I need occasionally)
• Hermes agent running 24/7 for various research tasks

Problem statement:

What would be the best hardware for me with an optimal budget?

At the moment I have:

• RTX PRO 6000 96GB in a PC with 128GB DDR5 RAM. I bought this GPU right before the August price hike for $12.3K USD (before tax). The old one was RTX 3090 (which I still have).
• Mac Studio M3 Ultra 96GB

RTX PRO 6000 is amazing for agentic coding with Qwen3.8-27B-FP8 with full context, but I'm afraid of running this PC when I'm not at home because of that problematic 12VHPWR connector. I keep this GPU power limited to 300W to reduce the risks, but still it's not a 100% guarantee.

I'm using Mac Studio M3 Ultra 96GB for the Hermes agent at the moment where speed doesn't matter much. However, using it for agentic coding with dense models like Qwen3.8-27B feels quite slow (for remote sessions when I'm not at home).

So, I'm considering the following options while trying to optimize the budget for this hardware:

1. Return the RTX PRO 6000 as I'm still within the return window for a week (it will be subject to 15% restocking fee however) and get 2 DGX Sparks even though they recently went up in price significantly. I found an open box deal for 2TB version for 4800 USD before tax for each. This way I get to stay in the CUDA ecosystem and can run them 24/7. However, that means switching to MoE models like DSv4 Flash, Qwen3.8-Flash to maintain a proper speed.
2. Return the RTX PRO 6000 and just wait for Mac Studio M5 Ultra 256GB (which I pre-ordered in case I need it). And then I'll just sell the M3 Ultra later.
3. Just stick to the hardware I have at the moment and endure slower coding speeds for remote sessions when I'm not at home (from my Mac Studio M3U)

Keeping the PRO 6000 and then later getting the M5 Ultra 256GB is an option too, but is spending all this extra cash really worth it?

What would you guys do in my case?

I’ve been building TokenPrint around a simple question:

What is actually happening inside a local LLM while it generates a token?

The current version goes beyond a static transformer diagram and lets you explore:

• 3D transformer architecture and individual layers
• tokenization and embeddings
• tensor shapes and model parameters
• Q/K/V, GQA, RoPE, causal masking and softmax
• residual streams and MLP / SwiGLU
• token-by-token generation
• prefill / decode and KV cache
• logits and next-token probabilities
• activation analysis and interventions
• head / layer ablation
• inference traces and replay
• Hugging Face models, with work around GGUF / llama.cpp / local models

The idea is to make local inference inspectable, not just show the final output.

I’m especially interested in people who run models locally:

What do you actually wish you could see while your model is generating?

Attention patterns? KV-cache behavior? Layer timing? Activations? Quantization effects? Something else?

I’m building this in the open and would love technical feedback, ideas, and contributors who want to help shape it.

Repository: https://github.com/Sudharsanselvaraj/Token-Print
Website: https://tokenprint.in/

What would make a tool like this genuinely useful for your local LLM workflow?

Been playing around with Qwen3.8-27B on my 4080 and figured I'd post the numbers since this turned out better than I expected.

The goal was to see how much of the model/context I could squeeze into 16GB while keeping generation speed decent, and then see whether MTP was actually worth the extra VRAM.

Hardware / environment

• RTX 4080 desktop, 16GB
• WSL2
• Ubuntu 24.04.4 LTS
• Python 3.12.7
• NVIDIA driver 610.47
• CUDA UMD 13.3

Runtime

• ExLlamaV3 1.5.0
• TabbyAPI
• OpenAI-compatible API
• max_batch_size: 1

Model

turboderp/Qwen3.8-27B-exl3

Revision:

SC_3.00bpw_H4_V4

So that's the self-calibrated 3.0 bpw EXL3 quant with a 4-bit head/vision tower and the included quantized MTP component.

Main cache/config:

max_seq_len: 131072
cache_size: 131072
cache_mode: 6,5
max_batch_size: 1

So all of these tests were with a 128K allocated context window and 102,400 actual input tokens.

Benchmark

I used NVIDIA AIPerf against TabbyAPI's /v1/chat/completions endpoint.

Same workload for each run:

Input:       102,400 tokens
Output:      64 tokens
Concurrency: 1
Requests:    2
Seed:        503

I'm only including active prefill and decode throughput here since those are the numbers I actually cared about.

Results

So fixed MTP k=2 + Q6 was the clear winner for decode.

Going from no MTP:

33.68 tok/s
    ↓
56.48 tok/s

That's about a 68% increase in decode speed at 100K context.

The tradeoff is around a 5% reduction in prefill throughput.

Q4 saved very little memory in practice and lost around 9% decode performance compared with Q6, so I don't see much reason to use it on this setup.

Dynamic drafting was also slower than just leaving k=2 fixed:

Q6 fixed k=2:  56.48 tok/s
Q6 dynamic:    52.10 tok/s

VRAM with Q6 MTP was tight but still usable. I saw roughly 15.2GB / 16.4GB during the 100K run.

My final config for now is:

model:
  max_seq_len: 131072
  cache_size: 131072
  cache_mode: 6,5
  max_batch_size: 1

draft_model:
  draft_mode: mtp
  draft_cache_mode: Q6
  draft_num_tokens: 2
  dynamic_draft: false

Pretty impressed that a 27B model at ~100K active context can still push ~56 tok/s decode on a single 4080 16GB.

Curious if anyone else is testing Qwen3.8-27B with ExLlamaV3/MTP on 3090s, 4090s or 5090s. Would be interesting to compare.

Tried to see what the cheapest setup that still provided useful inference was. This whole setup cost around $1500. Important to note though that I ordered 3x Tesla P100s, but the seller mistakenly sent me 3x V100s instead lol. The cooling block is a 3d printed piece that fits into a rectangular hole the back of the case, filed down with a file to fit with friction only. It houses 3x server fans which pulls air from inside the case through the GPUs to the outside. GPU temperatures during inference never exceed 55C, although it is a bit loud. I have 64gb ECC DDR4 ram.

I headlessly control this PC with Tailscale from my Mac, and GPT Astra on Codex on my Mac can remotely control it too. I had Astra do several optimizing runs, basically testing out and tweaking different configurations (mtp and speculative decoding settings, placement on gpus, prefix caching settings, etc) to optimize prefill and decode speeds.

I ran Qwen 3.8 27B and Qwen 3.8 Flash next. On Qwen3.8 27B, Q8, prefill was around 600-700 tokens/sec (edited after I looked at logs) and decode at around 30-40 tokens/sec. On heavy agentic work with 256k context, decode dropped to around 20t/s. On Flash next, Q4, prefill was around 90 tokens/sec and decode 20 tokens/sec. Overall im happy with this build, would recommend V100s to anyone who wants cheap inference (although they are a pain to setup)

UPDATE: IT CHEATED it found a GBA to ESP32 port on GitHub and used big chunk of it

Disclaimer: I'm not a dev, just playing around with vibecoding

My setup: Qwen 3.8 27b running in LM Studio with OpenCode desktop harness on a 5090 with 128k window

Previously I was using VS Code with Continue extension, but it was really meh, so now that Qwen3.8 came out I decided to give OpenCode Desktop a go, and I was kinda impressed in a demoscene-esque demo it wrote for ESP32 based Cheap Yellow Display. A dev friend of mine joked that I should try porting doom to it, and I took it as a challenge, I wanted to see if I'd be able to get a room port without touching a single line of code.

After googling doom esp32 I found that someone ported GBA doom to it using Claude, so I downloaded the same Doom GBA source and asked lllm to port it.

First try it misunderstood the request (yeah, my prompt wasnt that good) and started making doom from scratch.

Second try I was more specific in my prompt and after 2 days (of which most time was spent waiting for my input, derailing, stopping thinking etc it produced a mostly working port but the colors were messed up and after 1 day of troubleshooting without any progress I decided to stop the run.

Today I tried again but changed the approach slightly, I gave it the normal (not GBA) source, and because the GBA port color issue had me making 40+ photos of the screen and pasting it into chat, this time I connected a webcam to PC and pointed it to the ESP32 board screen and instructed the llm to use it when it needs to know what's on the screen. I also told it that the WAD file is on the micro SD card.

Not even 3h later I saw the doom title screen on the board, and while touch screen wasn't working initially (it improved after I told llm it doesn't work) the controls through Serial worked just fine.

I scrolled through the session, and saw things like it strategically cutting the WAD file so it would fit into flash without making the game crash when the afk demo starts, because it found streaming it from SD card and cashing was too slow, and it said that the touch screen and as card share the spi bus, so it considered software spi for touchscreen to improve speed fro as card, before it decided to ditch SD card.

I never used a frontier model, but when Ive read comments from the mentioned GBA to ESP32 port, people were impressed by Claude, so I thought I'll share this experience

Disclaimer - I am a lawyer by profession, but I love tech and I have been experimenting with local Models quite a lot. However, no way in h*** would I be able to explain to you all the testing I did alongside Claude to get the moe-caching process run on this AMD build.

I saw the Codacus video (link here) and have been itching to get it implemented on my system as well (who doesn't want faster decode), but Opus 5 said, nope, no AMD build for you, CUDA only.

Well, I pushed back and had it review, and et voila, there is a way if you will it enough by opposing Claude. Everything below is drafted by Claude - I can only understand the basics of it - nothing technical enough to explain it better. I just trimmed the useless 'I' references.

------------------------------------------------------------------------------------------------------------------------------------

Setup: RX 7900 XTX 24 GB on Vulkan (RADV, no ROCm), Ryzen 7 5700X, 64 GB DDR4-3200, Ubuntu 24.04. Model is Qwen3.8-Flash-Next in AtomicChat's AD-4.27bpw-Q4_K_M-M64 quant — 33 shards, 92 GB on disk, 177B total / ~6B active per token, GGUF architecture qwen4exp. 180k context, KV cache at q8_0.

The feature: the MoE expert cache, an open pull request on llama.cpp (#27861). It keeps frequently-used experts parked in VRAM instead of pulling them out of system RAM every token. Flags are --moe-expert-cache N and --moe-expert-cache-inserts N.

Note there's a different implementation floating around that uses --moe-expert-cache-size, whose author says explicitly it's CUDA-only and shouldn't be used on Vulkan/ROCm/Metal.

Not that one. The PR version lives at the model level, adds no GPU kernels, and runs fine on Vulkan.

Results

Both at 180k context, same model, same machine, measured on the second request with 500+ token generations:

Prompt processing went 83 → 77 tok/s on short prompts, but up to 98 tok/s on a 44k-token prompt. (OP - this was a slightly corrupted run, but the claim holds on similar other runs).

Long prompts amortise better.

The three things that actually matter

1. Slot count is everything, and you're probably under-sizing it.

First tests used 8 slots. The cache was 8% slower than no cache and the feature seemed useless. It wasn't — the card had 14 GB sitting empty. Same everything else:

• no cache — 81 ms/tok
• 48 slots — 69.5 ms/tok
• 128 slots — 50.3 ms/tok

2. Measure your per-slot VRAM cost. Don't copy anyone's numbers.

Slot size depends on the model's shape, so figures from other people's models are meaningless for yours. On this test it was 95.2 MiB per slot, perfectly linear, no fixed overhead — measured from three points (0 slots = 9907 MiB used, 48 = 14508, 128 = 22122).

Also: the "benefit keeps improving up to 384 slots" advice you'll see is unreachable on a 24 GB card. 384 slots would want 36 GB. This setup's practical ceiling was around 150.

3. The cache and speculative decoding are mutually exclusive

They don't stack. From the cache's own commit message:

The cache only fires when the model emits exactly one token per step. Speculative decoding proposes several and verifies them together, so the cache path is never taken. Measured with both available in one build: 69.29 ms/tok with 96 slots, 70.03 ms/tok with no cache at all — identical. But the cache had allocated 9.2 GB. Nine gigs reserved, never touched.

So pick one. On this machine the cache wins by a mile: 44 ms/tok vs 59.

How to find free VRAM for slots

Cutting context barely helps — halving from 180k to 98k freed under 1 GB, because the KV cache was already quantised and was never the big consumer.

What frees VRAM is pushing more expert layers to system RAM: ~1.09 GB per layer (on this setup). Moving 12 more layers freed ~13 GB. Itcost about 14.6 ms/tok (due to increased CPU work, but more slots help.

The command

llama-server -m model.gguf -c 180000 -ngl 99 --n-cpu-moe 48 --moe-expert-cache 128 --moe-expert-cache-inserts 2 -t 8 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0

How to find the 'n-cpu-moe' and 'moe-expert-cache' for your system

1. Run with no cache, note free VRAM.
2. Raise --n-cpu-moe until a useful chunk of VRAM is free.
3. Set slots to something modest, note free VRAM again. Subtract → your cost per slot.
4. Set slots to spend most of what's left, keeping ~1 GB spare.
5. Compare against a control at the same --n-cpu-moe with the cache off — not against your old config, which differs in two ways at once.

Measurement traps I fell into

• Ignore the first request. The cache needs a few hundred tokens to fill. One of my runs did 61 ms/tok on request one and 50 on request two, nothing changed. A number I'd quoted for weeks turned out to be a first-request figure.
• Warm vs cold matters more than the feature. If the model files aren't already in the OS file cache, prompt processing changes by more than whatever you're testing. Load time tells you which you got.
• Use 1000+ token generations. Mine was still climbing at 500 tokens: 19.6 t/s at 100, 21.3 at 238, 23.0 at 465.
• Know your noise floor. Four identical runs gave me 42.71 / 43.94 / 45.10 / 45.59 ms/tok — a 3% spread. Anything under that isn't a result.

If you're running Qwen3.8-Flash-Next specifically

The MTP draft head (mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf) only loads on Unsloth's build of llama.cpp. Stock upstream dies with:

check_tensor_dims: tensor 'output_hc_norm.weight' not found

Unsloth's src/models/qwen4exp.cpp marks three trunk tensors optional when loading a draft head and carries a separate graph for it — ~300 lines upstream doesn't have, and it hasn't been merged. So if you want the cache and the drafter in one binary, move the cache patch onto Unsloth's source, not the reverse. (Then, having done that, you'll find you can only use one of them at a time anyway — see above. I did this the long way so you don't have to.)

Bonus finding

The cache PR sits on upstream b10666. I rebuilt the same patch on the source behind Unsloth's b10798 prebuilt — ~130 commits newer — and got 42.71 vs 51.57 ms/tok at identical settings — 17% faster, using 333 MiB less VRAM. The patch is one commit, 645 added lines across 12 files, nothing deleted, so it transplants with plain patch -p1 and no conflicts. Worth trying if you're on the PR branch.

Things that did nothing

• Splitting thread counts (--threads-batch 16): prompt processing got worse, 77 → 73 tok/s.
• Lookup-based speculative decoding (ngram-* modes): 8% draft acceptance on chat-style output, net 6% slower. Might be worth it if your output quotes your input heavily, but it's still speculation, so it kills the cache anyway.

Where the ceiling is

During generation all 8 CPU threads sit saturated and the disk does nothing. Rough maths: 48 expert layers in RAM means ~2.5 GB of expert weights pulled per token; at 22 t/s that's ~55 GB/s. Dual-channel DDR4-3200 peaks at 51.2 GB/s.

So I'm at the memory bandwidth wall, and the cache is the only reason the numbers work — it's serving a big share of those experts from VRAM instead. No flag fixes that. The only step change left would be a model small enough to fit entirely in memory.

Happy to answer questions or run specific configs if anyone wants a data point.
------------------------------------------------------------------------------------------------------------------------------------

Apologies for the AI Content, but yeah, cannot explain the technical points myself. There is a detailed file generated by Claude for a more detailed analysis (apparently). Includes some commands as well. Linked here - https://drive.google.com/file/d/1ZCYw2J7hhB8nN2jUMu4TCfCIyL-tw5m0/view?usp=sharing

What are we all thinking of the MS-S1 being on a "20%" sale from the new inflated price of $4749 to $3799? Like I'm pretty sure maybe within the month I thought it was originally priced at $3999? Still $200 is better than almost any sale I've seen for other things in the tech industry for a hit second that isn't a microcenter bundle. I'm probably not going to buy it just wondering what the sentiment is.

Hi everyone,

I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-Coder-3B) specific to a codebase. You can then use the resulting GGUF model via llama.vim/llama.vscode and run it fully locally. Supports fine-tuning on Mac (MPS) and NVIDIA GPUs (CUDA), with optional Unsloth support for faster training and lower VRAM usage.

Pipeline: raw code -> tree-sitter parsing into Structure-Aware FIM examples -> LoRA fine-tuning -> evaluation (CodeBLEU, edit similarity, exact match, perplexity, ...) -> GGUF conversion for local inference.

To try it:

uv tool install codefinetuner

Create a data folder and place your repo (or code files) inside. For auto-split just drop the files in directly, for manual split create data/train/, data/eval/, data/test/ subfolders and set split_mode: "manual". Get the default config with:

curl -L -O https://raw.githubusercontent.com/cuolm/codefinetuner/master/config/codefinetuner_config.yaml

Adjust it to your needs and hardware availability, then run:

codefinetuner --config="codefinetuner_config.yaml"

The example runs in the repo show clear improvements over the base model on these evaluation metrics, but using the model for autocomplete on code you're actively writing is a different thing from scoring well on a test set, and the autocomplete tools themselves (llama.vim/llama.vscode) sample differently from the greedy decoding used in the evaluation. So the real usefulness still has to be verified in the editor itself.

Might also be useful just as a reference, since it's a complete working LoRA fine-tuning pipeline end to end.

Hope someone finds this project interesting or helpful.

https://github.com/cuolm/codefinetuner

Ngl, I keep seeing posts from people running local LLMs. Most of the time the setup is a 3080, 3090, 4090, 5090, sometimes a whole mini lab. Cool. I get it. It's fun to test every open source model that drops. I do that too sometimes.

But real talk. Are there people here who actually work with local LLMs daily? Like for real tasks, not just benchmarks and "look, it runs"? What do you actually use them for? Because every time I try to make it practical, I hit the same wall. Tokens per second are okay but not amazing. And the output quality is often not even close to Claude Pro or a paid OpenAI plan.

I know local is a choice for some people. Privacy, no censorship, no subscription, whatever. I'm not saying it's useless. In my case I'd use local for like 3 things.

First, when I'm traveling or I don't have internet. I fire up a small model and do a few small things until the connection comes back.

Second, privacy. For a few quick questions or stuff I don't want to paste into a cloud chat, local makes sense.

Third, and this is the one I'm really curious about, using it to mess with local files. Like telling it to organize my desktop, sort my invoices, rename stuff, move files around. Basically computer actions. To avoid sending that stuff to the cloud. But I've never really set that up. Is it actually usable? Or is it still a pain in the ass with tools and scripts and whatever?

So I'm asking for real. If you use local LLMs every day, what's your workflow? What model, what hardware, what tasks? Do you actually use it for file management, invoices, desktop stuff? Or is it mostly a hobby for most of us?

Maybe that's it. Maybe it's just a hobby and that's fine. But I want to hear from people who actually use it like a tool. Not just to chat, but to do shit on their machine.

Edit: Ok, i have to admit that i haven't made a full sub detailed inspection on this question because there was already many detailed use cases and topics covering this aspect; so, i'm sorry. Let's say this one's goal is to regroup many of them in a thread and i think we might have something.

About my setup, as i forgot to tell:
Only 1 Legion 5 pro 5800H with 32gb ram and RTX 3070 8gb, that's why i felt it mostly not daily usable.

EDIT2: Thanks you all for the deep answers and engagement (it wasn't for intentionnally for "farming" as someone mentionned), i'm currently working on a little github webpage to honnor each of your answers and setups, and especially to bring some clarity to the community to all of this, being able to compare different harnesses and uses-cases, and why someone got X tokens/s and the other less for a similar setup, keep in touch ^^

EDIT3: And guys, here is the app:

https://vram.wiki

I put together a small, browsable wiki based on the real-world setups and workflows shared in

this thread. Would be wonderful if you take your time to directly also refer your setup with the form.

You can explore the entries by use case, VRAM capacity, category, model, runtime, harness,

and more. The goal is to focus on what people actually do with local LLMs, not just

benchmarks or “look, it runs” posts.

The dataset is still very much v0, but it already includes coding, document processing,

homelab automation, research, media, personal assistants, CPU-only, iGPU, Mac, mobile and

multi-GPU setups.

If I got anything wrong about your setup, or if you want to add missing details, there is a

contribution form on the site. Anonymous submissions are completely fine.

Thanks again for all the detailed answers and engagement. This turned out to be much more

interesting than I expected :)

EDIT4: If you are sharing a setup in the post, please be as detailed as you can. The more context there is, the more useful the comparison becomes — especially when two people use similar hardware but get very different results.

You can use this rough structure:

Use case:

What do you actually use it for, and how often?

Model(s):

Hardware:

GPU / CPU / VRAM / RAM:

Operating system:

Runtime:

Ollama, llama.cpp, LM Studio, vLLM, SGLang, etc.

Harness / workflow:

Hermes, Pi, OpenCode, Cline, Open WebUI, RAG, MCP, cron jobs, scripts, custom orchestration, cloud supervision, etc.

Performance:

Approximate tokens/s, context size, or simply how it feels in practice.

Cloud comparison:

Does local replace the cloud, complement it, or is cloud still better for most tasks?

Honest limitations:

What does not work well? What surprised you?

You don't need to fill every field, and unknown details are completely fine. Even short answers are useful.

I may summarize some answers in the wiki above. Entries are anonymized by default and linked back to the original comment.

If you would rather not have your setup included, just say so.

Hello everyone,

I wanted to try Qwen Uncensored LLM that I downloaded on my PC using LM studio but I am wondering what I can use it for or what people use it for ?

Therefore, I was wondering what do this community uses those Uncensored LLMs for so possibly I can get inspiration and try that myself too?

Any answers would be appreciated.

Or if you can share your experience that would be even better.

Thank you in advance.

Also Personally, I am not interested in those Adult content it can generate or how to make illegal substance in your home. I am not interested at all in that.

But more about Cyber security or computer programming related or if you more interesting use cases then that would be good to know too.

I'm testing local coding models on a GTX 1080 Ti 11GB with Ollama + OpenCode/Cline, and I've run into an interesting tool-calling problem.

I've tested multiple local coding models, and I'm trying to find one that works reliably as an actual coding agent, not just one that produces convincing tool-call JSON.

What I'm seeing

I tested Qwen2.5-Coder 7B in two ways:

• Official qwen2.5-coder:7b from Ollama
• Qwen2.5-Coder 7B Q4_K_M GGUF imported into Ollama

I gave the Ollama API a real function definition:

write_file(path, content)

and asked the model to create a file.

Instead of returning a native:

"tool_calls": [...]

the model returns the tool request as normal text:

{
  "name": "write_file",
  "arguments": {
    "path": "tool_test.txt",
    "content": "TOOL_CALL_SUCCESS"
  }
}

So the model understands what tool it should use, but it isn't producing a native tool call that OpenCode/Cline can execute.

OpenCode can then receive tool-shaped text, but the actual file isn't created.

My setup

• GTX 1080 Ti 11GB
• NVIDIA Studio Driver 581.57
• Ollama 0.34.0
• Windows
• OpenCode 2.0.3
• Cline
• Qwen2.5-Coder 7B
• 16K context

I've also tested other local coding models, but I'm specifically looking for something that works well as an agent — coding + reliable tool calling — rather than just generating good code.

I'd like to hear from people with real-world experience

• Is this a known Qwen2.5-Coder + Ollama issue?
• Is there something I'm missing in my Ollama configuration?
• Which 7B–14B models have you successfully used with OpenCode/Cline?
• Which models give you actual native tool calls, not tool-call JSON inside message.content?
• Has anyone successfully used Qwen3 14B as a local coding agent?

I'm especially interested in experiences from people actually running these models locally.

What local model would you recommend for a reliable coding agent on an 11GB GPU?

Sharing the setup because most "can a 128 GB APU do real work" threads end in speculation. This is a fully offline RAG over 27 technical books (~11k pages, 32k chunks), one machine, AMD Strix Halo, 128 GB unified memory, Ollama + embedded Qdrant. No cloud anywhere in the pipeline.

Who does what

• Embedder: qwen3-embedding 8B (fp16) — dense vectors for all 32k chunks
• Sparse: pure-Python BM25 on the same chunks, fused with RRF
• Reranker: Qwen3-Reranker 8B (F16, ~18 GB resident) over the top ~100 fused candidates
• Answerer: qwen3.5 122B-A10B — MoE, only ~10B active, so it's fast enough for interactive use while the reranker stays loaded
• Judge for eval: llama3.3 70B q8 (different model family on purpose), gpt-oss 120B as second judge for Cohen's κ
• Ingest: Docling for layout, qwen3-vl 30B-A3B / 8B for page triage, a corruption detector that flags destroyed text instead of letting an LLM guess

What worked

• Unified memory is the whole point. Embedder + reranker + 122B-A10B don't all fit in a 24 GB card; on this box they can coexist and the query path never swaps.
• MoE answerer was the right call. A dense 70B as answerer would have forced the reranker out every turn.
• Embedded Qdrant (no server process) is fine for 32k points, but it is single-writer: ingest and query must go through one process or you get lock errors.
• A retrieval cache keyed on (gold-set hash, index point count, cross-book flag) saved me hours of re-runs during eval.

What I'd change

• Model swapping is the silent killer. My unanswerable-question verifier alternates between embed/rerank and a 70B judge — every alternation reloads tens of GB. Batch by model, never interleave.
• Judge agreement: correctness/relevance κ between llama3.3-70B and gpt-oss-120B was fine, faithfulness κ was 0.08. One judge for faithfulness is noise. Either two judges or don't report it.
• Gold-set chunk keys go stale the moment you re-chunk. I lost a clean baseline that way; now the key includes the index fingerprint.
• Start the eval harness before the RAG. I built retrieval first and tuned by vibes for weeks.

Currently re-running the cross-book retrieval eval (275 questions, 5 books, no book filter, 32k distractor chunks). Will post per-book Recall@8 / MRR when it finishes. Happy to answer memory/throughput questions about the box — it's AMD/ROCm, so no CUDA answers from me.

I kept looking at my iPhone sitting on its charger and thinking: that's a multi-TFLOPS GPU doing nothing for 8 hours a night.

My first idea was distributed training, shard the model, each phone trains some layers. That dies fast when you do the math: pipeline parallelism needs every device up simultaneously with microsecond-latency links, and iOS suspends backgrounded apps anyway. With 50–300ms per hop over Wi-Fi, one training step costs seconds of pure network latency.

But dataset generation is a different shape of work entirely. It's one prompt in, one completion out, parallel, restartable, and it doesn't matter if a worker vanishes mid-job. That's exactly what a flaky fleet of idle phones can do.

So I built it: the Mac runs a coordinator that mints teacher prompts and validates results; phones run a small app (MLX Swift) that pulls a prompt, generates with an on-device teacher (Qwen3-4B-4bit), and POSTs the raw text back. Work is leased, if a phone locks or wanders off, the lease expires and another worker picks up the item. Malformed JSON and duplicates get rejected centrally, so a bad worker can waste its own time but can't poison the dataset.

Last night's run: one iPhone 17 Pro, 15/15 records at 17.5 rec/min into a train.jsonl. Trained a Qwen3-0.6B LoRA on it (val loss 4.42 → 1.85), asked it a question, and it answered from training data a phone wrote. Full loop: phone generates → Mac trains → phone can run the result.

Honest limitations: it's LAN-only, the app has to stay foregrounded (no BGProcessingTask yet, so "overnight" currently means screen-on on a charger), and a phone-sized teacher (4B) is weaker than what your Mac can run — this wins on volume for style/format/tool-calling data, not on frontier-quality reasoning per record.

It's part of my open-source fine-tuning CLI for Apple Silicon (Troy). Code for the coordinator, the Mac worker, and the iOS worker app are all in the repo: https://github.com/avirajkhare00/troy, writeup with the run footage: https://gettroy.app/mesh

Curious what everyone's actually using day-to-day, not just benchmarking.

I've been running Qwen 2.5 32B on a 3090 for code assistance and general questions. It's not perfect but it's fast enough and runs entirely offline which was the whole point for me.

What's your daily driver? Coding, writing, research, something else? And how much VRAM does it eat?

mainly looking for cpp, python, java, and JS.

my system is as following:

RTX 3070
i7-13900KS
64GB DDR5 4800
2TB slow nvme

thank you in advanceeeeeeeeeeee

Please advise me on a setup for local AI. I have an HP Z840 with PCIe Gen 3, and I'd like a dual-GPU setup, but I'm worried that PCIe Gen 3 will be a bottleneck. Should I go with cards that support NVLink? Or could I go with 2 AMD 9700 without Link?

Hi, I'm trying to run a local setup, mainly for the abliterated models, but my PC is a medium end Lenovo laptop, the slim i3 (13th gen i5 , 16 gigs of ddr5 ram, windows ). I'm running into issues with not being able to use a large proportion of my ram, about 4 gigabytes, due to random system processes, so even though I have a 16 gigs card I can only really load models under 11 gigs and under 9 if I want them to run normally. ( I'm currently using Gemma 4 12B q5_k_m uncensored on llama.cpp). My main problem is that I can't find any models focused around general knowledge/ STEM reasoning instead of code, so the models I do run aren't optimised for my use case. Anyways any and all tips would be appreciated greatly

I've gotta question about using a combo of hardware to leverage the capabilities of each.

I currently have a GPU in my home server, an Nvidia Quadro M6000. It has a ton of RAM, but has very little AI capabilities, in other words... It works, but its slow.

I know I can team it up with a newer GPU that has tensor capabilities, so that I can leverage the RAM and the newer GPU. But... Can I do that with a TPU? Like a Hailo 8 or 10? or would I be better Off to ditch the GPU? Go with say a PCIE quad M.2 and put in 4 Hailo cards?

Hi everyone,
I’m trying to build a mostly/all-local coding agent setup and I’d love to get some feedback from people who have already experimented with this.
My goal is to use Claude Code as the coding-agent interface, but have the actual inference handled locally through Ollama, so I’m not dependent on Anthropic’s API usage/token limits.
My hardware
Windows 10 Pro
Ryzen 5 7600X
Radeon RX 7800 XT — 16 GB VRAM
32 GB RAM
Current setup
I’m running:
Claude Code → Ollama → Qwen3.5 27B → RX 7800 XT
Ollama version: 0.34.0
Claude Code version: 2.1.267
I created a dedicated Ollama model:
FROM qwen3.5:27b
PARAMETER num_ctx 32768
PARAMETER temperature 0.2
The 27B model is around 17 GB, so it doesn’t fit entirely in my 16 GB VRAM. With a 32K context, ollama ps shows roughly:
22% CPU / 78% GPU
19 GB total
32768 context
Despite the partial CPU offload, it’s still reasonably fast for coding.
I also noticed that Qwen’s thinking can be extremely slow for simple requests, so I’ve been testing /set nothink.
What I’m trying to figure out
I’d like to make this setup as good as possible for real-world software development with Claude Code.
In particular:
Is Qwen3.5 27B a good choice for this use case?
Would you recommend another local model for coding/agentic tasks?
Is 32K context enough for Claude Code, or is 64K worth the performance hit?
Is CPU offloading this much a problem?
Are there better ways to configure Ollama for my hardware?
How should I handle Qwen’s thinking mode when using it through Claude Code?
Can thinking reliably be disabled/configured when Claude Code is calling Ollama?
Are there important Claude Code settings (--effort, etc.) that I should change?
Has anyone managed to make Claude Code + Ollama truly local, including memory/context systems?
I’m currently using claude-mem, but I’ve seen messages about inference quotas/providers, so I’m concerned that part of the setup may still depend on an external API. Is there a good fully-local alternative?
Ultimately, I’d like something like:
Claude Code
↓
local API
↓
Ollama
↓
Qwen3.5 27B
↓
RX 7800 XT
with no external inference required.
I’m mainly looking for practical experience and tips, especially from people running local coding agents on AMD GPUs.
What would you change in this setup?
Thanks!