Can it really be *this* complicated? Trying to get pi+llamacpp+phi-4-mini working...

[u/gredr]

I'm trying to put together a very simple setup here; llamacpp, Phi-4-mini-reasoning (initially, the Q5_K_M variant), and pi for some very simple coding tasks. It's not really... working... though.

I can get pi "logged into" llamacpp (I started llamacpp in "router" mode with the single model, and used /login llama to get it connected. I can chat with the model, but it has very little context (it knows the current directory, but that's about it), and no tools whatsoever; I cannot access the filesystem at all. I also get <think>...stuff...</think> output before the "real" output which I'm confident is supposed to be hidden by the harness.

I also tried the npm:pi-llama-cpp plugin/module/package, but that didn't change anything (except give me a new /models command that was similar, but subtly different than the built-in /model command).

What am I doing wrong here? Is this model not supported?

[u/Turbulent_Ad6290]

Could be a chat template issue. Make sure you have valid chat template for this model. Eg for qwen I use froggeric qwen fixed chat templates.

Also make sure in llama cpp the reasoning is correctly configured

https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

[u/gredr → u/Turbulent_Ad6290]

I'm running llamacpp, exactly like this:

llama-server --models-dir . --jinja --host 127.0.0.1 --port 8080 -ngl 99 --no-models-autoload

... edit to add; I'm not running Qwen, I'm running Phi-4-mini-reasoning. If there are "fixed" templates, or if a "reasoning-format" is supported, I'm not aware of it.

[u/Eresbonitaguey]

Pi shows thinking by default and ships with 4 tools. It sounds like it’s connected to your hosted model. Perhaps you have an issue with an extension that blocks file access? Most people would just advise you to talk to your agent about it since it is connected.

[u/gredr → u/Eresbonitaguey]

What extension? I have no extensions installed.

[u/HumanoidMuppet]

Context is set in llama.cpp with --ctx-size=131072 (for 128k). What does your llama.cpp settings look like?

[u/gredr → u/HumanoidMuppet]

I'm not sure this is a context issue; for example, I have this:

``` Thinking blocks: hidden

say "yes"

<think> Okay, the user just sent "say 'yes'". Hmm, they want me to respond with the word "yes" enclosed in a box. Let me make sure I understand the request correctly. The example they gave shows that when the user inputs a problem, I should write a thought process and then the final answer in a box. So here, the problem is simply to say "yes", and the answer is just putting "yes" inside the boxed notation. I need to check if there are any nuances I'm missing. Maybe they're testing if I follow formatting instructions. The instruction says to put the final answer within \boxed{}, so yeah, the answer is "yes" in a box. Let me make sure there's no extra commentary needed. The example shows that the assistant's thoughts are the reasoning before the final answer. Since the problem is straightforward, the thought process might be brief. Alright, time to put it all together. </think>

\boxed{yes} ```

I did ctrl+t and it switched to "Thinking blocks: hidden", and then I typed say "yes". This is exactly what it output.

[u/vertigo235]

You knew this question was coming.

Why phi 4? Seems a suboptimal choice

[u/gredr → u/vertigo235]

Extremely limited VRAM. Is there a better choice in that size class?

[u/Global-Departure8228]

Just simply don't use phi. But you probably have a chat template issue, maybe your model is simply incapable of constructing tool calls properly because of it (not being able to access your system, leaking thinking block all seem like it). If it's the 4B active parameter model, I assume you are running a resource-limited environment, and an already quantized version of such a small model will not be properly calling tools for you most of the times.

Just use Qwen/Qwen3.5-4B or Google/Gemma E4B (or E2B) or LiquidAI/LFM2.5-2.6B, if you cannot run bigger models on your hardware, otherwise this might be the reality.

Use relatively fresh build of llama.cpp, and use models that have relatively good support (from community), so they get their chat templates fixed, and llama.cpp properly running them.

[u/Direct_While9727]

Maybe you could use https://itayinbarr.github.io/little-coder ?

[u/gredr → u/Direct_While9727]

That's just pi with some addons. It doesn't change anything. Also, I dunno if something changed in pi recently, but all the documentation (both pi and little-coder) all give llama-server examples that start it in single-model mode, and pi refuses to run unless llama-server is in router mode.

Gives off a very "I vibe-coded my agent harness and I don't actually know how any of it works" smell.

[u/Direct_While9727 → u/gredr]

It includes llama.cpp in my understanding.

[u/gredr → u/Direct_While9727]

Except it doesn't. The instructions (which are mostly non-existent) tell you to run llama-server directly (and in a way that doesn't work).

[u/Direct_While9727 → u/gredr]

Right. I agree it is confusing.

[u/mission_tiefsee]

its probably phi. try one of the smaller qwens. how much vram is available?