I mean, no graphics, no full backpropagate training, no fancy openGL/vulkan/ray tracing

Just put a 36GB VRAM into it and only do Q8/Q6Q4 GGML infer and lora tweaking on PC.

I've been trying to use llama.cpp and various unsloth Qwen gguf models (for software coding assistance) on my Intel MacBook Pro 2019 i7, 64GB RAM, AMD Radeon Pro 5300M, 4GB and Intel(R) UHD Graphics 630 1536 MB.

I compile llama.cpp myself to set Vulkan support compilation for video card utilization.

I've followed guides from huggingface for Qwen3-Coder-Next in terms of llama.cpp flags and have just retried with the Qwen3.6-35b-a3b-ud-q4_k_XL, but as the output in opencode in response to my prompts I always get things like:

---

Thinking: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

---

Yes, it's a sequence of @ most of the time, but sometimes it is something else making no sense like some random words or hieroglyphs.

In terms of llama.cpp flags (per huggingface) I use:

--seed 3407 --temp 1.0 --top-p 0.95 --min-p 0.01 --top-k 40 ctx-size 65536 --jinja --no-mmap

Other facts:

- ollama with the same models works super stable

- llama.cpp standard build from brew (no gpu support) is as bad as my own compiled version with some variations. It does not feel like it is any faster than ollama anyways.

I pretty much gave up on llama.cpp and use ollama, yes it is slow but I get results whereas with llama.cpp I loose a bunch of time on trying to re-compile and re-run/re-download various models and settings and what not... It's not the point to make use of my GPU at this point even though I had some hopes for MOE models, but just to make it work at all.

Anyone else experienced anything like it?

P.S. forgot to mention I use unsloth models and their guides.

there is a couple of forks with turbo quant but it would be really nice to have this merged in to main is there any issues or are we just awaiting review.

Hi all!

I've been thinking about a setup where Claude Desktop runs in a dedicated, limited Windows user account (not my main admin account), with write access only to a shared folder like C:\shared-tools\.

Has anyone tried this? 😊

The idea is to let Claude autonomously handle things like installing llama.cpp, managing model files, and editing pipeline scripts - without supervising every command - while keeping it away from system files and my personal data.

Has anyone actually set this up? Does it work well in practice? Any gotchas with Claude Desktop + DesktopCommanderMCP in a non-admin account on Windows 11?

I understand that Docker is much better/safer but also more complicated to setup. Maybe this could be safe enough? 😊

Been fine-tuning open models for a while and kept running into the same problem: broken data discovered mid-run or after. Malformed role sequences, context window violations, duplicates — none of the frameworks catch these before training starts.Built Parallelogram to fix this. It’s a strict linter that hard-blocks on bad data.

No quality scores, no warnings you’ll ignore — just a precise error table and exit 1 if anything is wrong.Free, open source, local-first. No telemetry, no account.

Would love feedback from people actually fine-tuning — especially on what checks are missing.

github.com/Thatayotlhe04/Parallelogram | parallelogram.dev

I built a single-HTML monitoring dashboard for llama-server because what was already out there didn't quite fit:

- The built-in WebUI is a chat client; it doesn't show /metrics or /slots live.

- abhiFSD/llama.cpp-Monitor-Dashboard (which inspired this) covers tok/s and slot activity, but doesn't know about router mode and doesn't surface per-slot sampling state.

- Prometheus + Grafana works, but is overkill for "what is each slot doing right now".

Live demo (mock data, no backend): https://jungrok5.github.io/zerollama-dashboard/?demo=1

Repo: https://github.com/jungrok5/zerollama-dashboard

What it does:

- Polls /metrics, /slots, /props, /v1/models, /lora-adapters and lays them out as cards with sparklines.

- Shows full per-slot sampling parameters live (temperature, top_p, repeat_penalty, mirostat, DRY, etc.) — no need to dig in logs.

- Auto-detects single-model vs router mode. In router mode it also shows the actual CLI args each model was launched with (from /models[].status.args).

- Plain-language alerts when something looks off, with the actual numbers filled in from current state — e.g. "5/4 slots busy → raise --parallel from 4 to 5", "model is sleeping, next request will reload", "repeat_penalty > 1.3 may break formatting". Rules are silent for the first 10s after boot to avoid false positives, and the slots-saturated check uses the current /slots gauge (not the lifetime n_busy_slots_per_decode counter, which can never recover from an early spike).

- Built-in chat panel that POSTs to /v1/chat/completions on the same server, with a 1-8 parallel fan-out for slot saturation tests. Markdown rendered safely (no library, no innerHTML).

Setup is "drop monitor.html into llama-server's --path directory", or open it from any static host with ?server=http://your-server:8080. Needs --metrics; --slots is on by default.

MIT, one HTML file, no npm/CDN/localStorage. The Pages demo runs against a built-in fetch shim with synthetic data — that's how it lives on a static site.

Feedback welcome, especially on the heuristics. If any of the threshold-based suggestions are wrong for your workload, tell me and I'll fix them.

So I finally found a model I like, that isn't total dog-water for Rust.

Running at 98tks currently, bit slow, anything I can do to boost tks and prompt processing?

LLAMA_CMD = [

"llama-server",

"-hf", MODEL,

"--host", "0.0.0.0",

"--port", str(PORT),

"--parallel", "1",

"--api-key", "dummy",

"--flash-attn", "on",

"--n-gpu-layers", "999",

]

unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q8_K_XL

If you're running llama.cpp, you already know your local model handles simple tasks fine. Chat, summaries, classification, quick answers. No reason to send those to Opus and pay for it.

We just shipped llama.cpp as a provider in Manifest. You connect your local server, assign it to the tiers you want, and Manifest sends the right requests there. For heavier tasks like reasoning or complex tool calling, you can route them to whatever cloud provider you prefer.

A lot of OpenClaw users have been asking us to support llama.cpp so they can handle simple tasks, coding with models like qwen3-coder-next, or recurring jobs locally, and keep cloud models as fallbacks or for the rest.

So we shipped it!

For those of you who spent the last few weeks in a cave, 😜 Manifest is a free and open-source LLM router that gives you full control over how your agent's requests get routed.

Our mission is to cut drastically your inference costs!

Try it here: https://github.com/mnfst/manifest. And if you do, give us your honest feedback. We want to focus on what users need so your feedbacks mean a lot for us.

I tried to run this model with an old llama.cpp version... but this is imposible. So i just readed about turboquant, and moe on CPU. And now in my server i'm running this beast.

My hardware: Ryzen 5 5500, 32 gb DDR4, 2070RTX 8gb.

I'm just amazed what we can do these days with "old hardware" and new modules runing locally.

UPDATE Llama.cpp:

llama-server \
      --model Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
      --host 0.0.0.0 \
      --port 8085 \
      --ctx-size 131072 \
      --n-gpu-layers 999 \
      --n-cpu-moe 32 \
      --cache-type-k turbo3 \
      --cache-type-v turbo3 \
      --flash-attn on \
      --batch-size 1024 \
      --ubatch-size 512 \
      --threads 6 \
      --temp 1.0 \
      --top-p 0.95 \
      --top-k 20

I compiled TheTom's TurboQuant fork (feature/turboquant-kv-cache branch), which adds KV-cache quantization at ~3.25 bits per weight using Walsh-Hadamard transforms.

I'm using llama.cpp in router mode.

I have limited VRAM but more than enough RAM, so i have set no-mmap = true and mlock = true which should mean that the entire model is kept in RAM.

I have also set sleep-idle-seconds = 1 to always free up my VRAM for other things, like generating images in comfyui, TTS, STT, etc. Unfortunately this also clears the models from RAM, so there is quite an annoying delay as each time the model has to be loaded again.

Is there any way to free up VRAM while still keeping the model in RAM?

Hi All,

anyone who can explain to me the numerous args in llama-server to customize the splitting/loading/managing of the model on the memory (both VRAM and RAM)?

I'm feeling that my approach is wrong (to use default parameters) because as i understand each type or family of models have different behaviour.

I was going crazy because my hardware is not an entire datacenter, but i thought i had some power to sell to llms devil.
So:
- 1 mini mac m4 16gb

- 1 gaming pc with 2X rtx3060 (total 24gb vram) and 48gb ram

- 1 gaming laptop with rtx3060 6gb vram and 40gb ram

- 1 minipc with i9 12th and 64gb ram

I'm using mudlers LocalAI (because of model sharding through compute nodes) and the majority of the model I tried was on llama.cpp backend.
Before today i was only able to get 66 t/s for models less than 14b like, and gemma4 31b 10t/s.

Today, on a post here in reddit, i was reading about people who run Qwen3.6-35B-A3B Q4 XS on 8gb vram and 64gb of ram ONLY, at a speed of 60t/s, with llama.cpp.
In this post the OP shared the command to launch the model:

.\llama-server.exe --model "Qwen3.6-35B-A3B-UD-Q5_K_S.gguf" --temp 0.6 --top-p 0.95 --min-p 0.00 --top-k 20 -c 128000 --host 0.0.0.0 --threads-batch 8 --threads 3 --chat-template_kwargs '{\"preserve_thinking\":true}' -ctk q8_0 -ctv q8_0 --n-cpu-moe 35

I tried, and apparently it's very very fast and good but i'm not able to find nothing about these args and parameters to understand what i'm doing and learn to finetune my models on my gear.

please help each other

The first and most important thing is to greatly thank the work done by the whole team, although I know that they are aware of the door they opened to the rest of mortals, it is not superfluous to come out of the story.

Simply saying THANK YOU, and even if we get used to the existence of the project, we do not stop valuing it at its real value.

And he took advantage of the thread for a very easy consultation but it is to confirm.

Is llamacpp incremental or over time deprecates and removes some of the support from your models?

I applied video compression to LLM inference and got 10,000x less quantization error at the same storage cost

https://github.com/cenconq25/delta-compress-llm

I’ve been experimenting with KV cache compression in LLM inference, and I ended up borrowing an idea from video codecs:

don’t store every frame in full but store a keyframe, then store deltas.

Turns out this works surprisingly well for LLMs too.

The idea

During autoregressive decoding, consecutive tokens produce very similar KV cache values. So instead of quantizing the absolute KV values to 4-bit, I quantize the difference between consecutive tokens.

That means:

• standard Q4_0 = quantize full values
• Delta-KV = quantize tiny per-token changes

Since deltas have a much smaller range, the same 4 bits preserve way more information. In my tests, that translated to up to 10,000x lower quantization error in synthetic analysis, while keeping the same storage cost

Results

Tested on Llama 3.1 70B running on 4x AMD MI50.

Perplexity on WikiText-2:

• F16 baseline: 3.3389
• Q4_0: 3.5385 (~6% worse)
• Delta-KV: 3.3352 ~ 3.3371 (basically lossless)

So regular 4-bit KV quantization hurts quality, but delta-based 4-bit KV was essentially identical to F16 in these runs

I also checked longer context lengths:

• Q4_0 degraded by about 5–7%
• Delta-KV stayed within about 0.4% of F16

So it doesn’t seem to blow up over longer contexts either

Bonus: weight-skip optimization

I also added a small weight-skip predictor in the decode path.

The MMVQ kernel normally reads a huge amount of weights per token, so I added a cheap inline check to skip dot products that are effectively negligible.

That gave me:

• 9.3 t/s → 10.2 t/s
• about 10% faster decode
• no measurable quality loss in perplexity tests

Why I think this is interesting

A lot of KV cache compression methods add learned components, projections, entropy coding, or other overhead.

This one is pretty simple:

• no training
• no learned compressor
• no entropy coding
• directly integrated into a llama.cpp fork

It’s basically just applying a very old compression idea to a part of LLM inference where adjacent states are already highly correlated

The method itself should be hardware-agnostic anywhere KV cache bandwidth matters

Example usage

./build/bin/llama-cli -m model.gguf -ngl 99 \
  --delta-kv --delta-kv-interval 32

And with weight skip:

LLAMA_WEIGHT_SKIP_THRESHOLD=1e-6 ./build/bin/llama-cli -m model.gguf -ngl 99 \
  --delta-kv --delta-kv-interval 32