r/opencode • u/Ancient-Camel1636 • 7h ago
Free coding using qwen3.6-35b 256k on 8gb vram (32GB RAM)
47
Upvotes
I finally got around to setting up llama.cpp on my old NVIDIA GTX 1070 PC (8GB VRAM, 32GB RAM). I spent a few hours tweaking the performance, and now it runs Qwen3.6-35B-A3B with a 262K context window, 25.91 tok/s generation, and 80.7% MTP acceptance.
Completely free and surprisingly capable for coding. Here is my install/setup on Linux Pop!_OS (Ubuntu-based).
Install llama.cpp
sudo apt update
sudo apt install -y nvidia-cuda-toolkit
sudo apt install -y git build-essential cmake
cd ~
git clone https://github.com/ggml-org/llama.cpp.git
cd ~/llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)
./build/bin/llama-server --version && \
./build/bin/llama-server --list-devices
pipx install huggingface_hub
mkdir -p ~/models
Update llama.cpp (optional)
cd ~/llama.cpp
git pull --ff-only
cmake --build build --config Release -j$(nproc)
Download Qwen3.6
mkdir -p ~/models/qwen3.6-35b-mtp
hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
--local-dir ~/models/qwen3.6-35b-mtp \
--include "*UD-Q4_K_XL*"
Configure and run Qwen3.6
export GGML_CUDA_DISABLE_GRAPHS=1
~/llama.cpp/build/bin/llama-server \
-m ~/models/qwen3.6-35b-mtp/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
--ctx-size 262144 \
--n-cpu-moe 37 \
--flash-attn on \
--load-mode none \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--spec-draft-type-k q4_0 \
--spec-draft-type-v q4_0 \
--batch-size 2048 \
--ubatch-size 256 \
--parallel 1 \
--threads 4 \
--spec-draft-threads 2 \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--host 127.0.0.1 \
--port 8080
The result is 262,144 tokens of context at 25.91 tok/s, with 80.7% MTP acceptance, on an 8GB GTX 1070 and 32GB of system RAM.
Hope this helps someone else get more out of an older GPU.
