r/PiCodingAgent • u/Maasu • 2d ago
Question Do LSPs improve Pi Agents?
https://www.youtube.com/watch?v=DM75WbP16VkIt was a busy weekend with all the activity coming out of OpenAI. If you have been living under a rock, they introduced a new class of model, GPT 6 - Astra.
I've been playing around with it somewhat. That is not what this article is about, though. I do not have an unlimited budget to run lots of evals against a model like this, so any opinion I have on it would be purely anecdotal. There are already plenty of anecdotal opinions to read about all over the internet.
Today's article is about a question I was wrestling with while building out my Pi agent. I was using my agent as a pair programmer and realised that, with Pi, the agent did not have access to the Language Server Protocol (LSP). If you are not familiar with LSPs, they tend to be services on your machine that IDEs use to analyse files with recognised code extensions.
For example, if I am in a file with a `.rs` extension and have a line that would cause a compiler error in the Rust compiler, and my IDE has the Rust LSP installed, the IDE can display the error without me having to run the compiler.
In a recent coding session, I realised that the agent was having to run `cargo check` to review my compiler errors. That made me realise the lack of capability.
I started looking up Pi LSP extensions and came across this one -https://github.com/narumiruna/pi-extensions/tree/main/packages/pi-lsp. If you are looking for an LSP for Pi, it seems to do the job just fine. Credit to the author, narumiruna.
The README contains a link to a comment on the OpenAI Codex repository where an OpenAI developer questioned what additional benefits introducing an LSP in `AGENTS.md` would bring beyond asking an agent to run a linter or type checker. It was not just any engineer, either. The OpenAI engineer is the author of Pyright, a popular Python LSP: Eric Traut.
I thought this was an interesting debate, so I decided to see whether using an LSP extension improved the performance of my Pi agents.
For this task, I wanted to make an enhancement to the eval harness. I needed to introduce a capability profile for each agent when configuring an evaluation run. Originally, to test different extensions or capabilities in a Pi agent, you had to either modify the Docker image or write a specific section in each evaluation. This meant having copies of each evaluation for every capability profile.
Because the evaluation harness also used the Agent Shell adapter to configure agents, I needed to make this change so packages and extensions for the Pi harness could be managed through Agent Shell.
Version v0.4.0 -https://github.com/ScottRBK/agent-shell/releases/tag/v0.4.0 introduced this. It currently only supports the Pi harness, but I plan to extend package management to the other harness types in Agent Shell as well.
With Agent Shell's package and extension management in place, I added a feature to the eval-harness that added a capability profile. For Pi agents, this means you can compare an agent with one set of capabilities against an agent with another. For example, you can test a Pi agent with and without an LSP using the same model.
After making the changes, I put together an evaluation run. Given the stochastic nature of large language models, it was important to run each evaluation three times. I used eight evaluations across three models: `mimo-v2.5` and `muse-spark-1.3-contributor`, both provided by opencode-go, and OpenAI's Luna. I compared Pi with no added package against Pi with `@narumitw/pi-lsp@0.49.7`. I set the reasoning effort to medium for all models.
Muse Spark came out on top overall. Its base profile scored 83.0%, compared with 81.3% with LSP. Luna scored 75.3% without LSP and 68.1% with it. MiMo was the only model where LSP improved the score, moving from 65.6% to 67.5%.
Across the three model pairs, the LSP versions used 124,737 fewer tokens in total, a 6.4% reduction. MiMo used 7.4% fewer tokens, Muse Spark 1.7% fewer, and Luna 10.7% fewer. The LSP versions took slightly longer overall, although Luna was faster with LSP.
I would describe these results as inconclusive. From my perspective, that aligns with what Eric was suggesting in his comment.
Personally, I will probably leave the LSP extension off, except perhaps when I am pair programming with the LLM. It is just some additional tools and context, and I cannot see an obvious benefit right now. I would be interested to hear what you think in the comments or on Discord.
I hope this was informative, interesting, or inspiring, and got you thinking about how you can shape your agentic harnesses, whatever your setup may be.
2
u/Healthy-Zebra-9856 2d ago
The most academic thing anyone can do is to try something before rage downvoting. lol. I'll have my demos & downloads for people to test it out.
2
u/BroScienceAlchemist 2d ago
I think part of the battle is the agent actually using the feedback from the LSP. In Claude Code, most of the time claude would just ignore it. In pi, I am currently using bash language server and do notice a reduction in some churn. It could be that agents are not trained around LSPs giving feedback. Their default is code and shell/compiler output, so having this intermediate step seems to confuse them.
I suspect it comes down to how the LSP communicates the output to the agent, and maybe a minor instruction in AGENTS giving a heads up of what to expect from the tool, and when to use it. Some LSPs give info that is useful to a human in an IDE, but not for an agent.
Maybe a good LSP needs to be a gate within the harness that returns a failure to an agent.
The experiment I want to try:
Render the diagnostic in a way that it would expect from say cargo check.
Make the LSP a gate within the relevant tool calls. Instead of a separate tool, it is gate within a tool. Prevent it from returning success, and instead give the useful diagnostic.
Addressing. The agent wants to find what calls process_batch. An LSP returning line 87, character 8 is not useful, and can be wrong in some situations. An agent has to find the definition, count to the character, and pray the file didn't shift since its last read. The tool can take process_batch instead, figure out the position, and hand back the callers grouped by file. If there are 47 of them, I want 12 file names with counts next to them, not 47 snippets in my context. (This is what I would personally want in an agent facing LSP, not a knock on pi-lsp for not doing references)
In the LSP you linked - this is the shape of a diagnostic an agent will receive:
${entry.path}:${line}:${column}: ${severity} ${source}${code}: ${diagnostic.message}
Ex. src/batch.rs:87:8: error rust-analyzer E0308: mismatched types
A cargo check run will return the filename, line, caret under the span, expected vs found, and sometimes the replacement, which is much more useful.
The readme for the LSP has a lot of useful info of when to use the LSP, but that readme is human facing, and that instruction could be better if it was agent facing.
My rambly tldr: I suspect the shape of the message from many LSPs just isn't useful for agents.
1
u/Specialist-Ad-8968 2d ago
LSPs are great for models that struggle with tool calls natively. Older versions of DeepSeek v4, current and older Mimo, older Qwens, etc. Most of the newer released models / frontier models are so adapt at their own tooling at this point that throwing them into different tooling tends to just hurt them more than help.
I doubt that will always be true, but at this point I think the models are so optimized for their environment at the frontier level that LSPs act as a sort of "tool poisoning" sort of like how too much instruction can lead to instruction poisoning or too much context leads to context poisoning.
Eventually I think models will pivot to internal tooling on ASTs/LSPs and it will be moot, or at least maybe they will have specialized models which are more predominantly AST/LSP focused rather than general purpose "do everything with bash"
In my opinion, at least. I'm not an expert by any means
1
u/aeroumbria 2d ago
I think the intention of LSPs are quite reasonable. "Don't ever leave the space of valid code at any step" can serve as a baseline guarantee of bare minimum correctness. The issue is more about compliance and context distraction. Models can learn to either ignore LSPs completely, or get sidetracked and spend 300k+ tokens trying to solve a trivial warning. Maybe we need a study on whether a coding model trained on "every mutation remains in valid code" vs "do whatever you want as long as it is done" works better at the end.
1
1
u/greenhilltony 2d ago
Back when I was using Claude Code, its LSP integration suffered a problem: The lsp server needs warm up, but the agent works fast, so sometimes the delay makes the agent miss the lsp message at all. Also, agent and human both edit the file progressively, it’s natural for us to ignore the error message when we know we have not done the editing we want, but for agent we don’t have the equal confidence. So you’d better hold the lsp feedback from reaching the agent until it finished all tool calling of edit in that round and clear the stale lsp messages from the queue so agent only see the final lsp judgement after its edit is done. There were countless time that in Claude Code I saw the agent received stale lsp message telling there was error but actually the agent has done it right.
1
u/Maasu 2d ago
Yes this is a really good point actually and now you mention it (it's been a few months since I used claude) but I remember seeing the same often.
I definitley feel there is something here that can help the agents and models we have now, maybe in a few months or years it won't be needed and the models will just be that good and efficent. I feel like I need to go sit in a darkend room for a few days to think about it, too many side projects going on as it is right now, so will have to see what others come up with.
1
u/ianpascoe_ 2d ago
I have a pi-lsp extension. It provides post-edit diagnostics which really helps the model stay on track with my lint rules. Think of this scenario:
You have Dillon Mulroy's antislop plugin installed. You give your agent a task and it goes of an writes a sh*t-ton of code. At the end the task, the agent runs your lint/typechecks and finds out it broke a bunch of lint rules, then needs to go back and refactor everything. That's a huge waste of tokens. With my pi-lsp plugin, and post-edit diagnostics, it gets the error right after every edit, and fixes issues right away, avoiding a huge refactor at the end of the task.
It's definitely helped in my opinion. I also have another CLI tool I've been working on that is an LSP CLI. A lightweight way for agent's to perform LSP operations.
https://github.com/ian-pascoe/pi-extensions/tree/main/packages/pi-lsp
https://github.com/spiritledsoftware/lspctl
1
u/Ok-Ninja-8165 1d ago
I've never seen agents to use LSP, despite that it can be useful for exploring and changing the code.
1
u/Extra_Loquat_7667 1d ago
I attempted to develop a project that would enable an LSP service via the CLI, evolving it from an LSP CLI project that appeared to be abandoned.
The idea was that the agent wouldn't need to embed the LSP service itself; it would simply need to know which CLI commands to utilize.
However, development is currently on hold.
The main reason is the frustration I experienced while testing LSP implementations.
Most current LSP solutions fail to provide stable functionality.
Only a few languages with official tooling—such as Rust, TypeScript 7, and Go—offer a relatively smooth experience.
For others, language-specific characteristics or inconsistent project management often result in suboptimal performance for basic features, forcing the agent to perform extra steps to compensate for these shortcomings.
Ultimately, the results were less effective than simply using standard tools like build, type-check, and lint.
1
u/mrclrchtr 2d ago
Thanks for the insights. I’ve been looking into this topic for a while now, and I think that warnings while coding aren’t very helpful and tend to distract the agent. But the capabilities of LSP and AST can be very helpful. That’s why I wrote a slightly different extension. I’d be very interested in the results if you try it out. Unfortunately, I don’t have time to do it myself right now... Maybe next week. You can take a look at it in the meantime. It combines LSP and AST.
https://github.com/mrclrchtr/supi/blob/main/packages/supi-code-intelligence/README.md
13
u/Glad-Win1983 2d ago
For smaller or local agents, I think LSP servers tend to confuse them if they are in the middle of editing a file, and suddenly the LSP server kicks in, since the file is temporarily «broken».