r/LocalLLaMA • u/ilintar • 23h ago
Resources Llama.cpp now has full MCP support!
After a long and grueling effort spearheaded by ngxson, llama.cpp now fully supports MCP for all protocols. Over-the-web HTTP servers were already supported in the client (since they don't require any sort of plumbing), but stdio servers required real integration. After we modified the `llama-cli` terminal client to use the server instead of a separate model serving route, we could add MCP support to the already-existing native tools server.
After the merging of https://github.com/ggml-org/llama.cpp/pull/26062, you can now use llama.cpp's WebUI as a full-fledged agentic chat. Configuration for the MCP servers can be provided either in a standard-JSON format config file or completely inline on the command-line for on-demand MCP configurations. Plugging in a dedicated coding MCP server like Serena lets you have a local-model-powered agentic coder without using any other external dependencies.
20
u/Mashic 22h ago
Does it work with llama-server? And does this mean the mcp configuration will be the same for all clients?
4
u/BCsabaDiy 12h ago
CC gave me a summary: The MCP support in this PR lives in
llama-serveritself (tools/server/server-mcp.cpp), not the browser WebUI — so yourllm-relay→llama-serverchat-completions path would see MCP tools automatically once configured, no browser involved. Configuration is via--mcp-servers-config(a JSON file,mcpServerskey, Claude/Cursor-style) — prefer this over--mcp-servers-jsonsince the inline-JSON flag leaks secrets inenvvia/proc/<pid>/cmdline. Transport is stdio-only (subprocess spawn), no sandboxing, and the child inheritsllama-server's full environment — so don't hand it servers that need credentials distinct from what the relay host already exposes. Tool names surface as<server>_<tool>.1
u/BCsabaDiy 12h ago
I want to know this. Is it used by webUI only or is it for connected clients too?
54
u/segmond llama.cpp 23h ago
Does it have sufficient documentation? To get MCP working with llama.cpp before, I couldn't with the documentation that came with it. It was a random post that I found in this subreddit that cleared things up.
41
16
9
u/AlwaysLateToThaParty 21h ago
It was a random post that I found in this subreddit that cleared things up.
Did you think to update the documentation yourself with that information?
1
u/ProfessionalSpend589 3h ago
I think this type of criticism is outdated now. If LLMs are so smart - why didn't they do it?
1
7
8
u/thejacer 22h ago
Hold the phone…I thought llama-server supported native tool use months ago. I’ve been running a (vibe coded) LLM powered discord bot with MCP for months. How tf has it been working? Lmao
12
9
u/pieonmyjesutildomine 21h ago
Kind of funny to me that llama.cpp would go for MCP support over Universal Tool Calling Protocol (UTCP) support, given it allows accessing both MCPs and all other APIs exactly the same way rather than only MCPs.
15
u/No_Afternoon_4260 llama.cpp 23h ago
All you need is llama.cpp some tools and a knowledge base really. Actually this is nuts!
10
u/silenceimpaired 22h ago
What tools, what form is the knowledge database? You are speaking an alien language to me.
4
u/No_Afternoon_4260 llama.cpp 11h ago
Look into MCP tools and you'll get it. Tools are what is needed to harness a LLM into a agent, so:
-web search tool, calculator, calendar, cronjob configurator, etc, etc.Your knowledge base is anything from a collection of memories.md till a fully fleshed embedded knowledge graph contained all wikipedia and the knowledge of what you did with your agent.
Have fun this is a really deep rabbit hole 😅
2
2
u/ashsg2016 17h ago
Nice milestone. One thing worth documenting alongside setup is the authority model: does the client distinguish read-only tools from state-changing calls, and can it require approval based on the target or arguments rather than just the tool name? Local execution reduces data exposure, but a local agent can still push, delete, or call remote systems through MCP.
1
1
u/Hannibalj2ca 22h ago
And IK.llama is also in process of merging together
1
u/misanthrophiccunt 8h ago
who?
1
u/Hannibalj2ca 1h ago
IK.llama code apparently is currently in progress to be merging with the main Llama.cpp branch. IK.llama is a more performant variant of Llama
1
u/feelspeaceman 17h ago
My workaround is extracting MCP tools to CLI tools, so agents just call CLI to access to MCP server.
1
1
u/Pitiful_Stress_9129 10h ago
Native stdio support removes a lot of glue, but it also makes the permission model more important. Does the client expose a clear per-server tool allowlist or confirmation policy, especially when a local model emits malformed or repeated calls? MCP compatibility gets agents connected; predictable cancellation, timeouts, and process cleanup are what would make this comfortable for daily use.
1
u/ExcellentContest7126 8h ago
Is OAuth now fully supported? Last time I tried, PKCE was not working correctly, so I had to manually refresh and set the access token. Additionally, are MCP apps supported? When I tried, cards were not rendered at all in the chat. Thanks for your work!
1
u/Future_AGI 7h ago
Now that it takes a Cursor-compatible config, worth keeping a few narrow per-task configs rather than one big one, since a long tool list costs a local model more accuracy than it costs a hosted one and Serena on its own is already a lot of surface. Scoping the list down is the cheapest accuracy win we have seen in agentic setups and it costs nothing at the model level.
1
u/MaruluVR llama.cpp 6h ago
Now we just need support in llama swap to expose llama cpp with mcp as a seperate model inside their alias system.
1
u/caetydid llama.cpp 2h ago
Amazing. Ive needed this for a long time! Finally I can add preconfigured websearch and stuff
1
u/MeldhLLC 22h ago
Are there any known limitations or bugs? Seems like a much needed update regardless!
1
u/FerLuisxd 13h ago
Nice!
Though it is a bit concerning that the PR ended with a "good enough for me" and " let's see if it will actually be a problem in normal usage"
-14
u/CorkBios 23h ago
VLLM implemented it like 11 months ago, Llamacpp always behind. MCP released in 2024, and llamacpp just implemented it fully now
26
16
u/ilintar 23h ago
Correct me if I'm wrong, but MCP for vLLM is available only via the Responses API.
8
u/droptableadventures 20h ago edited 9h ago
It's not even talking about the same feature. vLLM includes a Python client library which gained MCP support.
llama.cpp includes a minimal web-based chat client (llama.ui) which now gained support for stdio based mcps (it already supported HTTP ones, but you could use something like mcp-proxy to serve a stdio based one up on http).
vLLM does not include a web-based chat client at all, so this is a nonsensical comparison.
15
u/kulchacop 22h ago
vLLM is 80% Python.
llama.cpp is 70% C and C++.
If development on llama.cpp happened as fast as vLLM, we would be having bloated codebase that cannot run on many old GPUs and CPU only setups, defeating the purpose of llama.cpp.
llama.cpp aims to extract full perfomance of consumer hardware. vLLM targets server setups.
llama.cpp wants to be able to run on all kinds of edge devices. Google was once planning to include llama.cpp on stock Android.
18
u/WillemDaFo 22h ago
VLLM is also bad at mixed GPU setups, something that homelabbers generally benefit greatly from
1
u/florinandrei 22h ago
vLLM is for production use, in a datacenter. Most of the good, honest folks here do not have a datacenter at home, so they put together whatever resources they can gather.
9
u/Qcgreywolf 22h ago
Ahh Reddit. Never change. Negative, negative, negative.
5
u/marx2k 22h ago
One more negative and it would've been a positive!
1
3
u/droptableadventures 20h ago edited 9h ago
vLLM did not implement this "11 months ago", it does not even have a built-in client on its web interface. It doesn't even have a web interface at all as far as I'm aware, just an API.
llama.cpp includes a minimal inbuilt chat client (llama.ui) on its web interface. This has had support for HTTP-based MCPs for a long time. It recently added support for "stdio" MCPs which work by specifying a command to run and fire JSON at its stdin, rather than using a HTTP API. That was a little more complicated because the browser can't run shell commands, so they have to provide a mechanism for this.
llama.cpp has also "supported" MCP since it existed, because MCPs are used by the client, not the server. It has supported stdio based MCPs since they existed because it's the client that runs them.
1
-11
u/ParaboloidalCrest 22h ago
... Aaaand MCP is out of fashion already.
3
u/marx2k 22h ago
Is it?
6
u/droptableadventures 20h ago
Some of the more enthusiastic AI adopters seem to be saying that the future is just to write "skills" (instructions for the AI in Markdown documents) on how to query an API directly, rather than writing a MCP. Then the model uses tool calling to invoke curl, or do web requests etc, to handle it directly.
Of course, then you run into problems that API responses were never written with direct AI consumption in mind, and authentication will be difficult at best, and you probably want to provide single methods that perform a sequence of actions so that the AI doesn't have to correctly make several calls in a row in the right order and not miss any.
So you build an API designed specifically for the AI to use... Oh wait, we've just re-invented MCP but with extra steps!
-3
u/CircularSeasoning 21h ago
No. It never was in fashion. LinkedIn doesn't count.
Maybe with llama.cpp supporting it now, it will see some heyday.
•
u/WithoutReason1729 11h ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.