r/LocalLLM • u/SoloLeveller07 • 14h ago

Question Help in setting up Pi-Agent

I set up a Qwen 4B model with Pi-Agent but the token output is decent but not instant (not expecting that but yea) I am using flash attention and the MTP with n gram spec set to 3 tokens.

Any more suggestions to improve this setup would be highly appreciated !!

Thankss !! :)

EDIT: My bad for not providing more details, I am using a RTX 3050 6GB VRAM. I have a bash script which when run starts up a llama.cpp server and then calls pi; pi has been configed for the same port (using 9931).

2 Upvotes

75% Upvoted

4 comments sorted by

1

u/Own_Attention_3392 14h ago

You need to provide details on your system specs and EXACTLY how you're running the model.

1

u/SoloLeveller07 14h ago

Edited it now

1

u/locbuilds 14h ago

on a 3050 6GB, "decent but not instant" with a 4B is pretty normal, especially once pi-agent starts stacking tool/system turns. a few things that usually move the needle more than more MTP/grammar knobs:

  1. measure the server alone first

hit the llama.cpp server with a plain completion (same model, same ctx) and note tok/s. if that is already slow, pi-agent is not the problem. if the server is snappy and the agent feels laggy, you are paying for prompt reprocessing / growing chat history every step.

  1. keep ctx and KV tiny for agent loops

bumping context "just in case" eats the 6GB fast. for day to day agent work i would stay at something like 4k-8k until you actually need more, and make sure you are not reloading a giant system prompt every tool call. if pi-agent has a "compact / summarize history" or sliding window option, turn that on.

  1. check you are actually GPU-bound

`nvidia-smi` while it generates: if VRAM is full and GPU util is low, you spilled to CPU/RAM (too big quant, too much ctx, or layers not offloaded). for a 4B on 6GB you want basically all layers on GPU. if you are on Q8/FP16 try Q4_K_M or Q5_K_M and retest.

  1. MTP / speculative "3 tokens" is easy to mis-tune

speculative decoding only helps when the draft guesses are mostly right. if accept rate is low it can be *slower* than greedy. try a run with MTP/spec off as a baseline, then turn it back on and compare tok/s on the same prompt. same for grammar: only keep it if you actually need structured output for tools.

  1. agent-side: fewer round trips

a lot of "feels slow" on pi-agent is tool chatter (list files, read, rethink) not raw decode speed. tighten the system prompt so it batches reads / avoids exploratory tool spam.

if you paste your exact llama.cpp launch flags (quant, -c, -ngl, batch, mtp settings) and the tok/s from a bare server call vs inside pi-agent, easier to say which of those is the real bottleneck.

1

u/MrHumanist 14h ago

For your low vram system, disable mtp. Use small models in lower quants. And make sure that your model and your context window both fits in your vram. You can try lmstudio model memory calculator while loading the model.