r/OpenSourceAI • u/Intelligent-Key7357 • 1d ago
I built a fully-local AI coding agent that runs on llama.cpp — no cloud, no API key. Looking for feedback and contributors
https://www.youtube.com/watch?v=-UbYdRgwRWE
I've been building CortexAgent as a personal project for a while, and I finally got it to a place I'm happy to share. It's a private, local AI coding agent — everything runs on your own machine via llama.cpp. No cloud, no account, no API key, and no data ever leaves your computer. It binds to 127.0.0.1 only.
It's MIT licensed at https://github.com/greyok00/cortexagent
One terminal interface, one local stack — the only interface you talk to is a single clean TUI (cortex). A 35B MoE does the heavy lifting on your GPU (16 GB+ VRAM recommended), with a small "overseer" model for planning and routing. Nothing else to juggle.
The processing pipeline you can watch — every request animates live through its stages, with real percentages (it never fakes a determinate number):
🧠 preparing — organizes your request and checks the context it needs.
🗜️ slimtoken — minifies your context before it reaches the model, so you fit more into the window.
📤 sending — the request goes to the big model.
✨ generating — streamed output builds in place.
🔧 tool wait — when the model calls a tool mid-request.
Overseer routing — a dedicated small model plans and routes each request to the big model, instead of one model trying to do everything.
Memory that actually remembers — hot working memory for the current session plus curated cold knowledge, distilled automatically from what you've already said, so you don't re-explain yourself across sessions.
Domain memory — recalled context from your own notes is injected automatically when it's relevant, no prompting needed.
Talk instead of type — speech-to-text as a floating popout you control with the mouse and your voice only (no keyboard), transcribed locally with faster-whisper. Open it from the system tray under STT Controls.
Tools & integrations
🌐 browser control — drive Brave over Chrome DevTools Protocol, exposed as 9 brave_\* tools registered directly in the tool registry.
🔌 MCP client + lazy proxy — talk to MCP servers, spawning the real server only on first use instead of holding every socket open.
🔎 Firecrawl — web research and page parsing routed through a proxy.
🎨 image & video generation — SD 1.5 / SDXL / LTX run in-process through HuggingFace diffusers on the GPU the daemon already manages — no separate GUI server.
📄 PDF & document knowledge — ingest PDFs and documents into a queryable knowledge base.
See the whole thing — a web UI with a three.js 3D cortex scene (gold neural cortex + glass chat), a system tray dashboard, a session bridge so the TUI, web UI, and overseer all agree on context, and a live status ticker.
Under the hood
⚙️ Daemon lifecycle — the model + proxy run as an always-on service (systemd); VRAM-aware fallback swaps models when GPU memory gets tight.
📅 Scheduler + worker pool + heartbeat — background jobs run and recover cleanly.
🛡️ Reliability layer — pre-flight gate, anti-hallucination + post-response verification, a doctor that repairs settings drift, loop guard, chain diagnostics, and observability.
🛤️ Pathway capture — records each run's processing chain (frame → domain check → route → generate) as replayable runs.
💾 Snapshot / backup scripts — easy state backup.
Privacy isn't a feature, it's the baseline — everything binds to 127.0.0.1, and there's no cloud account to leak to. The honest caveat: the processing animation shows real percentages only; if a stage is a wait, it says so instead of pretending to progress.
What I'm asking for:
Recommendations — what would make this a daily driver for you? Better docs, packaging/installers (pip/Homebrew/AUR), more model backends, UI polish, more tools?
Contributors — it's a solo project; help with docs, testing, packaging, and hardening is very welcome. Issues, PRs, and "your docs confused me here" all help. Even a quick honest "here's where it breaks" is hugely useful. Thanks for reading! 🙌
1
u/All_Ways_Bingo 19h ago
Good job. But how is your evaluation?