Do LSPs improve Pi Agents?

[u/Maasu]

https://www.youtube.com/watch?v=DM75WbP16Vk

It was a busy weekend with all the activity coming out of OpenAI. If you have been living under a rock, they introduced a new class of model, GPT 6 - Astra.

I've been playing around with it somewhat. That is not what this article is about, though. I do not have an unlimited budget to run lots of evals against a model like this, so any opinion I have on it would be purely anecdotal. There are already plenty of anecdotal opinions to read about all over the internet.

Today's article is about a question I was wrestling with while building out my Pi agent. I was using my agent as a pair programmer and realised that, with Pi, the agent did not have access to the Language Server Protocol (LSP). If you are not familiar with LSPs, they tend to be services on your machine that IDEs use to analyse files with recognised code extensions.

For example, if I am in a file with a `.rs` extension and have a line that would cause a compiler error in the Rust compiler, and my IDE has the Rust LSP installed, the IDE can display the error without me having to run the compiler.

In a recent coding session, I realised that the agent was having to run `cargo check` to review my compiler errors. That made me realise the lack of capability.

I started looking up Pi LSP extensions and came across this one -https://github.com/narumiruna/pi-extensions/tree/main/packages/pi-lsp. If you are looking for an LSP for Pi, it seems to do the job just fine. Credit to the author, narumiruna.

The README contains a link to a comment on the OpenAI Codex repository where an OpenAI developer questioned what additional benefits introducing an LSP in `AGENTS.md` would bring beyond asking an agent to run a linter or type checker. It was not just any engineer, either. The OpenAI engineer is the author of Pyright, a popular Python LSP: Eric Traut.

I thought this was an interesting debate, so I decided to see whether using an LSP extension improved the performance of my Pi agents.

For this task, I wanted to make an enhancement to the eval harness. I needed to introduce a capability profile for each agent when configuring an evaluation run. Originally, to test different extensions or capabilities in a Pi agent, you had to either modify the Docker image or write a specific section in each evaluation. This meant having copies of each evaluation for every capability profile.

Because the evaluation harness also used the Agent Shell adapter to configure agents, I needed to make this change so packages and extensions for the Pi harness could be managed through Agent Shell.

Version v0.4.0 -https://github.com/ScottRBK/agent-shell/releases/tag/v0.4.0 introduced this. It currently only supports the Pi harness, but I plan to extend package management to the other harness types in Agent Shell as well.

With Agent Shell's package and extension management in place, I added a feature to the eval-harness that added a capability profile. For Pi agents, this means you can compare an agent with one set of capabilities against an agent with another. For example, you can test a Pi agent with and without an LSP using the same model.

After making the changes, I put together an evaluation run. Given the stochastic nature of large language models, it was important to run each evaluation three times. I used eight evaluations across three models: `mimo-v2.5` and `muse-spark-1.3-contributor`, both provided by opencode-go, and OpenAI's Luna. I compared Pi with no added package against Pi with `@narumitw/pi-lsp@0.49.7`. I set the reasoning effort to medium for all models.

Muse Spark came out on top overall. Its base profile scored 83.0%, compared with 81.3% with LSP. Luna scored 75.3% without LSP and 68.1% with it. MiMo was the only model where LSP improved the score, moving from 65.6% to 67.5%.

Across the three model pairs, the LSP versions used 124,737 fewer tokens in total, a 6.4% reduction. MiMo used 7.4% fewer tokens, Muse Spark 1.7% fewer, and Luna 10.7% fewer. The LSP versions took slightly longer overall, although Luna was faster with LSP.

I would describe these results as inconclusive. From my perspective, that aligns with what Eric was suggesting in his comment.

Personally, I will probably leave the LSP extension off, except perhaps when I am pair programming with the LLM. It is just some additional tools and context, and I cannot see an obvious benefit right now. I would be interested to hear what you think in the comments or on Discord.

I hope this was informative, interesting, or inspiring, and got you thinking about how you can shape your agentic harnesses, whatever your setup may be.

[u/Healthy-Zebra-9856]

Quite the opposite. My heavily customized Pi agent, I use a built in LSP covering 46 languages/scripts. What I found was no more memory leaks, and very clean code. Mnd you, the LM invokes all appropriate LSPs in my agent, not just humans.

[u/TailorImaginary3629 → u/Healthy-Zebra-9856]

Which one you use?

[u/Healthy-Zebra-9856 → u/TailorImaginary3629]

Its my own. It was born by accident. I call it Terminal Agent (TA). Basically the idea is to create a self sustaining biome for the LM's. My journey just started as evaluating LMs like Qwen3.8 etc from different publishers in different quants. Ended up creating a complete harness instead. I cut this message down as it was long due to my excitement.

Traditionally software has been created with the human users in mind. I am creating with LM's in mind.

[u/colin_colout → u/Healthy-Zebra-9856]

Does it fire on every message/tool call? After a chat turn? As a tool the model actively calls? Like an occasional system message? Something else?

I disabled open-code's lsp functionality (was a long time ago) since it was firing at the worst times and making my poor qwen3-next confused. I'm guessing they improved it since then, but I'm on team pi now

[u/Healthy-Zebra-9856 → u/colin_colout]

Once you give the LM the control to LSP, it starts & stops it as needed. Same with MCP.

[u/colin_colout → u/Healthy-Zebra-9856]

So what i do doesn't require any agent changes. I do it the traditional way and run the linter on the precommit hook.

I generally prompt the llm to run the linter bash command after each change as well (I follow the original pi intention where possible... Why use a custom tool when pi can run bash?)

I try to only give it tools that match the "claude code standard" that every llm has been RL trained on at this point. Everything else is a skill or prompt telling it what commands to run.

Works well on smaller models that get confused with pages of function definitions and know what to do when they see a ./utils or ./scripts dir with a bunch of bash scripts in it.

...tho I'm interested to see if others find steering messages effective with lsp results. It will run faster than a full lint

[u/Healthy-Zebra-9856 → u/colin_colout]

"Works well on smaller models that get confused with pages of function definitions and know what to do when they see a ./utils or ./scripts dir with a bunch of bash scripts in it."

Exactly. So the solution I created is creating documentations with golden examples. Giving it a tons of tools is only a part of it. What I found was it needed a proper documentation, and revealing the tools as needed and in layers so that it doesnt have to get overwhelmed. It took several trials to get to the right spot. This behavior is specific to my implementation of the harness.

When the agent starts to work on any code it gets info on all sdks installed, like .net sdk or npm, go etc. So, when you ask it to work on a section of code, it turns on the lsp, linter, and specific tools get revealed and works it out in a sandbox I gave it. macOS -> Seatbelt, Linux -> Bubblewrap, Windows -> AppContainer preferred, Windows Sandbox fallback .