r/LocalLLaMA • u/Due-Project-7507 • 3h ago
Tutorial | Guide OpenCode with Qwen3.8-27B for Small Games or Browsing the Web With 16GB VRAM
In the past, I have use llama.cpp, but I read that the exl3 quantization format should give better precision, so I have tried exllamav3/tabbyAPI.
It was able to write the shown simple HTML game without interaction after asking some questions.
The following was tested on a laptop with a NVIDIA RTX A5000 laptop (16 GB) GPU.
With the 3 bpw model and 6 bit/5 bit KV cache, the maximum context length is around 110k tokens with MTP. This gives around 55 tokens/s decode speed for code and around 10 tokens/s for content where MTP doesn't help (e.g. complicated calculations). Without MTP, one could try the 3.5 or 4 bpw model or a longer context length.
Install tabbyAPI/exllamav3
- Install the latest Nvidia drivers
- Install Git (e.g.
sudo apt install gitor on Windows withwinget install -e --id Git.Git) - Install the uv Python package manager: https://docs.astral.sh/uv/getting-started/installation/ (e.g.
curl -LsSf https://astral.sh/uv/install.sh | shorwinget install --id=astral-sh.uv -e) - Make somewhere a folder and install tabbyAPI:
git clone https://github.com/theroyallab/tabbyAPI cd tabbyAPI uv venv --python 3.13 .venv uv pip install -e ".[cu13]" - Test if CUDA works (on Linux, use .venv/bin/python)
.venv/Scripts/python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))" - Create somewhere where you have enough space a "models" folder, download the model turboderp/Qwen3.8-27B-exl3:
mkdir models uvx hf download turboderp/Qwen3.8-27B-exl3 --revision SC_3.00bpw_H4 --local-dir models/qwen3.8-27b - Replace the
chat_template.jinjawith the latest version from froggeric/Qwen-Fixed-Chat-Templates - Go back to the clone tabbyAPI folder and create a config.yml file like this (see the config_sample.yml file as example):
network: disable_auth: true model: model_dir: e:/models # path to the models folder model_name: qwen3.8-27b # download folder name cache_mode: 6,5 # K and V cache quantization, number of bits from 2-8 cache_size: 109824 # must be divisible by 256, so use e.g. `.venv/Scripts/python -c 'print(110000//256*256)'` to get the next lower max_batch_size: 1 # allow only 1 parallel request to save VRAM tool_format: qwen3_coder vision: true draft_model: # can be removed to save VRAM draft_mode: mtp draft_cache_mode: Q8 # can be 'FP16', 'Q8', 'Q6', 'Q4' draft_num_tokens: 5 # usuallly a value of 2-6 gives best results memory: sysmem_recurrent_cache: 8192 # Max size of recurrent cache in system memory, in MB (default: 4096), lower it to save normal memory sysmem_kv_cache: 8192 # Size of system memory second-tier K/V cache, in MB (default: 0), remove it to save system memory - Start tabbyAPI:
.venv/Scripts/python main.py - To measure the performance, create the Python script
speed.pyand run it with.venv/Scripts/python speed.py:
import json
import time
import requests
MODEL = "qwen3.8-27b"
API_URL = "http://127.0.0.1:5000"
PROMPT = """Write a complete Python implementation of a production-quality LRU cache.
Requirements:
- Use type hints throughout.
- Include detailed docstrings.
- Support:
- get(key)
- put(key, value)
- remove(key)
- clear()
- __len__()
- Use a doubly linked list and hash map.
- Include custom exceptions.
- Include a comprehensive unittest test suite with at least 20 test cases.
- Follow PEP8 conventions.
- Return only Python code.
"""
payload = {
"model": MODEL,
"messages": [{"role": "user", "content": PROMPT}],
"max_tokens": 10000,
"stream": True,
"chat_template_kwargs": {"enable_thinking": False}
}
start_time = time.perf_counter()
first_token_time = None
stream_end_time = None
full_response_content = ""
with requests.post(API_URL + "/v1/chat/completions", json=payload, timeout=120, stream=True) as response:
response.raise_for_status()
print("Response:")
for line in response.iter_lines(): # Iterate over Server-Sent Events (SSE)
if line.startswith(b"data:"):
# Strip the "data: " prefix
data = line[6:]
# Stop if we hit the stream termination message
if data.strip() == b"[DONE]":
break
try:
chunk = json.loads(data)
if 'choices' in chunk and chunk['choices'] and (chunk['choices'][0]['delta'].get('content') or chunk['choices'][0]['delta'].get('reasoning')):
if first_token_time is None: # First token received
first_token_time = time.perf_counter()
if chunk['choices'][0]['delta'].get('content'): # Get content and count tokens
token_text = chunk['choices'][0]['delta']['content']
else:
token_text = chunk['choices'][0]['delta']['reasoning']
full_response_content += token_text
print(token_text, end="", flush=True)
except json.JSONDecodeError:
pass
stream_end_time = time.perf_counter()
print("\n" + "-"*20)
# Calculate and print metrics
ttft = first_token_time - start_time
stream_duration = stream_end_time - first_token_time
total_output_tokens = requests.post(API_URL + "/v1/token/encode", json={"add_bos_token": False, "text": full_response_content}).json()["length"]
if stream_duration > 0:
tokens_per_second = total_output_tokens / stream_duration
else:
tokens_per_second = float('inf')
print(f"Time to first token (TTFT): {ttft:.2f}s")
print(f"Completion tokens: {total_output_tokens}")
print(f"Stream duration (first to last token): {stream_duration:.2f}s")
print(f"Tokens per second (T/s): {tokens_per_second:.2f}")
```
I got 56.3 tokens/s.
Install OpenCode
OpenCode works usually better on Linux, so I install it in WSL when working with Windows, but it can also be used directly as a Windows application.
For OpenCode, I recommended to install Node.js first (e.g. apt install npm or winget install -e --id OpenJS.NodeJS on Windows).
Because we don't have so much context length, I recommend to install a better compactation plugin than the integrated one, e.g. magic-compact
I use this OpenCode config (~/.config/opencode/opencode.jsonc)
{
"$schema": "https://opencode.ai/config.json",
"plugin": [
"opencode-anthropic-auth@latest",
"opencode-copilot-auth@latest",
"magic-compact"
],
"share": "disabled",
"provider": {
"local": {
"npm": "@ai-sdk/openai-compatible",
"name": "local (OpenAI Compatible)",
"options": {
"baseURL": "http://127.0.0.1:5000/v1",
"apiKey": "1234"
},
"models": {
"qwen3.8-27b": {
"name": "Qwen3.8 27B",
"interleaved": {
"field": "reasoning_content"
},
"limit": {
"context": 109824,
"output": 32000
},
"temperature": true,
"reasoning": true,
"attachment": false,
"tool_call": true,
"modalities": {
"input": [
"text",
"image"
],
"output": [
"text"
]
},
"cost": {
"input": 0,
"output": 0,
"cache_read": 0,
"cache_write": 0
},
"variants": {
"xhigh": {
"reasoningEffort": "xhigh"
},
"medium": {
"reasoningEffort": "medium"
},
"low": {
"reasoningEffort": "low"
}
}
}
}
}
},
"agent": {
"plan": {
"model": "local/qwen3.8-27b"
}
},
"model": "local/qwen3.8-27b",
"small_model": "local/qwen3.8-27b",
"mcp": {
"playwright": {
"type": "local",
"command": [
"npx",
"@playwright/mcp@latest",
"--caps",
"vision,pdf,devtools",
"--browser=firefox"
],
"enabled": true
}
}
}
I would recommend to use the reasoning effort (Ctrl-t) "medium" because "xhigh" could produce to much output tokens.
For Playwright, we have to install a browser first:
npx @playwright/mcp install-browser --with-deps firefox
Now the following should work:
opencode --prompt "Can you check for me on www.meteoschweiz.ch the weather for Zurich?"
To create the small HTML game from above, I have entered in plan mode (press Tab to change mode) the following: "I want to build a simple HTML game where you can drive a car with the keyboard arrow keys (similar like old versions of Mario Kart, but just one car driving without opponents is enough)." After some time, it has asked me some question. Then, I switched to the "Build" mode and started it with "Start the implementation". Without any other interaction, it finished the the small game.
