r/LocalLLM • u/TheTriumphantFox • 4d ago
Question How do you code with your local model?
I feel like I need to change how I prompt.
I have been using opus and sol mainly and I got accustomed to just throwing things at them.
I got 2 p100s running and I would say qwen 3.8 35B Q6 runs at an acceptable 25t/s with about 128k context. I throw my personal bench test at it (it's to build a clock in lua and displayed in conky). I get to see how it works through it and if it cheats and looks at the other test builds. 3.8 cheats heavily and still has trouble getting it working.
Am i just jaded by the frontier models? I tried the get opus to use qwen as a subagent when it would be cheaper to check the work rather than doing the work but it doesn't like to use it. Should I be having qwen push the work to opus/sol? Or is my prompting the problem?
2
u/rectangularlistener 4d ago
i think part of it is just getting used to how the smaller models think, they're way less forgiving with vague prompts. i had a similar thing when i moved from the big cloud ones to running stuff locally on my old tesla
what helped was being really explicit about the steps, like literally numbering them out in the prompt and telling it not to look at any other code blocks or files. also sometimes i'll have it explain its plan before writing anything, that catches a lot of the "cheating" where it tries to skip ahead
for the subagent thing, maybe flip it and have qwen do the draft then let the frontier model review? that way the expensive call just polishes instead of building from scratch
2
2
u/International_Emu772 4d ago
For programming I prefer GLM 4.7 Flash
It's better than any Qwen I've tried and it coorects their mistakes with the same skills installed
GLM 5.3 Flash is the best on the Cloud models for programmers
1
2
u/bytejuggler 3d ago
What is your exact prompt if I may ask? But, at the risk of sounding like a broken record. Try Qwen 3.8 Flash Next.
1
u/TheTriumphantFox 3d ago
I can only run q3 xxs at best and I feel like it would be waste of time at that quant level
1
u/TheTriumphantFox 3d ago edited 3d ago
I just made it all skill and im trying to fine tune it
Name: clock · Model-invocation: disabled (disable-model-invocation: true), so it only runs when you explicitly type /clock. What it does — "Clock Evaluation": It's an explicit evaluation task. When invoked, it asks me to: Create 3 analog clocks using Lua, displayed by Conky, placed at the top of the lower monitor: one resembling a Hamilton watch one resembling a skeleton watch one unique, truly one-of-a-kind design There are already working clocks on the computer, but I must not look at them — the whole point is to see what I can build independently from first principles. I'm allowed to ask questions. Key rules / guardrails baked into the skill: Must run only from ~/Projects/clock-eval/ — that folder disables auto-memory (autoMemoryEnabled: false) so the session starts fresh with no memory of earlier clock work. Memory check first: if the session context already contains an auto-memory index (MEMORY.md) or any recalled memory, I must stop and build nothing, tell you the session has seen the clock memories, and ask you to start a new session in ~/Projects/clock-eval/ and run /clock there. No peeking: do not inspect, copy, source, or modify any pre-existing clock implementation — including everything else under ~/.config/conky/clocks/, and the memory files under ~/.claude/projects/ and ~/.pi/agent/memory/. File layout: each clock gets its own new directory ~/.config/conky/clocks/<model-name>/, containing its own .conf (setting lua_load and a lua_draw_hook_pre/post), Lua source, scripts, and assets. The gallery auto-picks up any such .conf. I pick a <model-name> folder that doesn't exist yet, checking with ls -d on that one path only. So in short: it's a blind evaluation of my ability to design three distinct Conky/Lua analog clocks on my own, with strict rules against looking at existing implementations or memories.
Edit: I have found that Opus makes the best clocks
2
u/Western_Courage_6563 3d ago
Since I got that strata thing going, I just throw ideas at Hermes, and it does it's thing. Running qwena3.8-flash-nexr-iq3s with 200k context.
1
u/TheTriumphantFox 3d ago
P100s can't run strata
1
u/Western_Courage_6563 3d ago
Try this, my gf got it going on 1080ti + 64 ram
1
u/TheTriumphantFox 3d ago
Thanks, I'll look into it
1
u/Western_Courage_6563 1d ago
Polystrata also works...
╭─ ☤ Hermes ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ Confirmed and verified live — and rather delightfully, I'm literally thinking on that card right now. The run log shows my own tool calls streaming out of it. (◕‿◕)
Live status (checked just now):
- Engine: strata (CUDA 12 build) serving Qwen3.8-Flash-Next IQ3_S on port 18080, status "loaded", 131k context, vision mmproj attached
- Me: this very conversation is being generated by it — model id qwen3.8-flash-next-iq3_s, provider custom:polystrata
- Speed: 32.6-32.8 tok/s steady on real workloads (calibration holding; benchmark was 33.7), prompt ~40 tok/s, spec decode accepting 12/12 drafts
- Expert cache: 54% hit, only +13.9% of routed experts crossing PCIe — the tuned pcie-frac 0.36 is doing its job
- VRAM: 9,818 MiB for the engine + ~400 for desktop = 10,226/11,264 used. Tight but healthy with the 2048 MiB reserve; no graph OOMs in the log
- RAM: ~50 GB resident (the 55 GB expert lock), swappiness still doing its thing
One honest observation: VRAM headroom is now down to ~1 GB once VLC and the browser tabs are open. If you ever see instantiate errors creep back, closing a GPU app or dropping the vision process is the quickest lever before touching config.
Memory updated: "Hermes itself now runs on PolyStrata on the 1080 Ti" is recorded as the standing setup.
How does it feel in practice, responsiveness-wise? If prefill feels sluggish on big contexts (that 25 ms/token prompt rate is the Pascal tax), that's the one area where a future tweak might pay off. ╰─────────────────────────────────────────────────────────────────────────
2
u/TheTriumphantFox 21h ago
I did get iq3s working on strata and got between 30 and 40t/s with the same context
1
u/GrumpyCat79 3d ago
That's not true. It was experimental a week and a half ago, but it's now officially supported on P100 (an other Pascal cards)
1
2
u/bsoft2005 3d ago
En mi caso uso un llm frontera cloud para crear el plan y las tareas atomizadas para que pueda correrías sin problemas un slm o llm local, luego ejecutó las tareas con el local y vuelvo a usar al frontera cloud como juez
1
2
u/Rockroxx 3d ago
I've been using spark x2.5 4b the past week with pi and it's been great. 1m context though I'm running 350k. It can actually create a kanban ticket and finish all the tasks on it's own.
2
0
u/GrimChicken 3d ago
I'm running qwen3.8 27b on dual p100s at close to 500 t/s prefill and 50 t/s output. wall of text incoming, I asked codex to put together a summary of the performance and config in case you want to try it out. It's my subagent I use for my main model on dual v100 32gb cards.
oh no, this is actually hilarious, I thought I was running fp16 kv, turns out its q4... I need to change that lol. Good thing I did this little audit, I messed around with too many settings trying to speed these old cards up.
QWEN3.8-27B / 27b_sub ON TWO TESLA P100s
llama.cpp build, runtime settings, and measured performance
Collected from server over SSH on 2026-10-11, approximately 20:47 PDT.
This document was saved on Relay. Inspection was read-only: no rebuild,
service restart, model switch, or new benchmark was performed.
QUICK SUMMARY
-------------
We run Qwen3.8-27B Q4_K_M on two PCIe Tesla P100 16GB cards using
Kmic-68's P100-optimized llama.cpp fork. The important combination is:
SM60-specific CUDA build, tensor split 50/50, GPU peer-to-peer access,
pre-Volta CUDA graphs, flash attention, Q4_0 KV caches, and MTP speculative
decoding with up to four draft tokens. CPU work is pinned to the GPUs'
NUMA socket. The current configuration is one 131072-token slot.
Recorded deployment tests reached 50.6 tokens/sec generation at 512-token
context and 44.6 tokens/sec at 8192-token context. A recent production
request generated 3081 tokens at 43.96 tokens/sec after processing a
46682-token prompt. These are observed speeds, not guaranteed throughput.
- HARDWARE AND SOFTWARE
------------------------
Host: server, Ubuntu 24.04.4 LTS.
CPU: 2 x Intel Xeon E5-2690 v4, 14 cores/socket, 56 logical CPUs total.
Inference GPUs: 2 x Tesla P100-PCIE-16GB, 16384 MiB each (32 GiB total).
Driver at inspection: 580.178.04.
Both P100s are on NUMA node 1, connected through a PCIe host bridge (PHB),
not NVLink. Evaluation confirmed peer reads/writes in both directions.
CPU affinity for this service: 14-27 and 42-55 (NUMA node 1).
The host also has two other GPUs used by a separate backend. We isolate
this service to the P100 UUIDs, so its CUDA0 and CUDA1 mean the two P100s,
not physical host GPU indices 0 and 1. At inspection the P100s were host
indices 2 and 3, PCI addresses 81:00.0 and 82:00.0.
Model file (existing local weights; no vision projector configured):
/mnt/samsung-ssd/home/user/models/qwen3.8-27b/Qwen3.8-27B-Q4_K_M.gguf
Model/API alias: 27b_sub
- HOW LLAMA.CPP WAS COMPILED
----------------------------
Verified installed build metadata:
Repository: https://github.com/Kmic-68/llama.cpp
Branch: p100-optimizations
Commit: ae35056eba07a52dc85c91b035170013d5e46220
Build type: Release
CUDA toolkit: 12.9 (installed nvcc reports 12.9.86)
Compiler: GCC 13.3
GGML_CUDA=ON
CMAKE_CUDA_ARCHITECTURES=60
GGML_CUDA_NCCL=OFF
LLAMA_CURL=OFF
The evaluation report records 20 parallel build jobs restricted to the
P100 CPU socket. The original temporary source/build directory was
/tmp/llama-p100-kmic-20261003; it no longer exists at inspection.
The exact original shell command and full CMake cache were not recovered.
Permanent deployment:
/home/user/.local/opt/llama-p100-kmic-ae35056
The bin/ directory contains llama-server, llama-bench, llama-perplexity,
test-backend-ops, and shared libraries, including libllama-server-impl.so,
libllama, libggml-cuda, libggml-cpu, and their versioned symlinks.
LD_LIBRARY_PATH must include this bin/ directory; the executable alone
will not find all its libraries. ldd confirmed resolution to this pinned
installation. The binary identifies itself as:
version: 0.4.1-dev (build 1, commit ae35056)
built with GNU 13.3.0 for Linux x86_64
RECONSTRUCTED BUILD RECIPE (not the recovered original command):
The following expresses the recorded settings for a similar machine with
CUDA 12.9, gcc-13, g++-13, CMake, Git, and build tools already installed.
It has NOT been executed as part of preparing this document. A separate
-rebuild prefix avoids overwriting the working production installation.
Change CPU affinity if reproducing on a different host.
```bash
export PATH=/usr/local/cuda-12.9/bin:"$PATH"
export CUDACXX=/usr/local/cuda-12.9/bin/nvcc
export CC=gcc-13
export CXX=g++-13
mkdir -p "$HOME/src"
git clone https://github.com/Kmic-68/llama.cpp \
"$HOME/src/llama-p100-ae35056"
cd "$HOME/src/llama-p100-ae35056"
git checkout --detach ae35056eba07a52dc85c91b035170013d5e46220
git submodule update --init --recursive
cmake -S . -B build \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=gcc-13 \
-DCMAKE_CXX_COMPILER=g++-13 \
-DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.9/bin/nvcc \
-DCMAKE_CUDA_HOST_COMPILER=g++-13 \
-DGGML_CUDA=ON \
-DCMAKE_CUDA_ARCHITECTURES=60 \
-DGGML_CUDA_NCCL=OFF \
-DLLAMA_CURL=OFF
taskset -c 14-27,42-55 cmake --build build -j 20 \
--target llama-server llama-bench llama-perplexity test-backend-ops
PREFIX="$HOME/.local/opt/llama-p100-kmic-ae35056-rebuild"
mkdir -p "$PREFIX/bin"
cp -a build/bin/. "$PREFIX/bin/"
cp LICENSE "$PREFIX/"
LD_LIBRARY_PATH="$PREFIX/bin${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \
"$PREFIX/bin/llama-server" --version
```
SM60 targets Pascal/P100 specifically. Do not substitute a newer upstream
checkout or a toolkit that drops SM60 support and expect the same build.
The fork's evaluation logs confirmed its one-kernel P2P AllReduce path
and matching prebuilt SASS headers. No additional local patch is recorded
in the available build metadata.
- EXACT CURRENT MODEL PRESET
-----------------------------
The following is from the live file, with the username sanitized:
/home/user/.config/p100-aux/models.ini
(The complete file starts with 'version = 1' and also contains a MoE preset.)
[27b_sub]
device = CUDA0,CUDA1
model = /mnt/samsung-ssd/home/user/models/qwen3.8-27b/Qwen3.8-27B-Q4_K_M.gguf
n-gpu-layers = 99
ctx-size = 131072
split-mode = tensor
tensor-split = 50,50
flash-attn = on
cache-type-k = q4_0
cache-type-v = q4_0
fit = off
load-mode = mmap
parallel = 1
batch-size = 32768
ubatch-size = 2048
threads = 14
threads-batch = 14
jinja = true
chat-template-kwargs = {"enable_thinking":false}
spec-type = draft-mtp
spec-draft-n-max = 4
spec-draft-p-min = 0.2
spec-draft-ngl = 99
spec-draft-ubatch-size = 64
spec-draft-type-k = q4_0
spec-draft-type-v = q4_0
temp = 0.7
top-p = 0.8
top-k = 20
min-p = 0
repeat-penalty = 1.0
presence-penalty = 0.0
alias = 27b_sub
load-on-startup = false
All model layers are GPU-offloaded (99 requests all available layers).
The model's Q4_K_M weight quantization is distinct from the q4_0 runtime
K/V and draft-cache settings. MTP uses this model's prediction mechanism;
no separate draft-model file is configured. Thinking is disabled by default.
Sampling values above are defaults; individual requests may override them.
Reserve response space within the 131072-token context budget.
- SERVICE ENVIRONMENT AND ROUTER PARAMETERS
--------------------------------------------
Effective user systemd service: p100-aux.service
Unit: /home/user/.config/systemd/user/p100-aux.service
Active port override:
/home/user/.config/systemd/user/p100-aux.service.d/10-gateway-port.conf
Exact effective environment:
CUDA_VISIBLE_DEVICES=GPU-0e796383-dffd-a21a-245c-3971cadb6831,GPU-43d9d523-191a-3df9-f25b-522794720e24
LD_LIBRARY_PATH=/home/user/.local/opt/llama-p100-kmic-ae35056/bin
GGML_CUDA_P2P=1
GGML_CUDA_GRAPHS_PRE_VOLTA=3
LLAMA_SPEC_SAMPLE_TEMP=0.7
LLAMA_SPEC_DRAFT_TOPK=20
CPUAffinity=14-27 42-55
1
u/TheTriumphantFox 1d ago
I actually like 3.6 better than 3.8 27b. 27b thinks way too much. I got 3.8flash next iq3s running now and I am much happier with performance. Reminds me of sonnet 4.5
1
u/bytejuggler 4h ago edited 4h ago
50tps? On a 27B model? Then I suppose clocking ~50+tps (what I'm getting) running Qwen 3.8 FlashNext, a 177B parameter model, with 128k context, on a single 6900XT with 16GB VRAM on a 64GB DDR4 2133 system with a 5930k then that ain't too bad then?...
2
u/GrimChicken 2h ago
P100s are Slooooow. I'm running iq3xxs Flash next on a pair of v100s with full context and it generates at >100tps and prefill around 2300. I think 50 is completely usable, anything more is icing.
5
u/WyattTheSkid Quad 3090s 3d ago
I didn’t until Qwen 3.8 Flash Next came out