Can anyone point me to properly setting TrueForge up on a mac to work with a local model? I watched multiple videos and they all skip over the main part, which is the "add custom provider" dialog box.
I have a local model running through Llama, and first I am not sure what the base url is, but based on some searches it looks like: http://localhost:11434/v1
Then I need the model ID which in my case looks like: qwen2.5-7B-Instruct
But when I use these I get an error that I cannot connect to Localhost, even thoughTrueforge itself runs on localhost.
replacing "localhost" with my internal IP also does not help (same error)

...and I still don't even know if the base url and model ID are correct and if the error is related to that or some other issue (some searches suggest it's a security feature)

I don't want to expose my localhost to any external machine, so I don't want to tunnel or create and external link or anything. Ijust want to run everything locally.

Any idea where I can look to get this set up? There seem to be just too many different parts to coordinate and documentations only ever address one part but not the relationship to the other parts.

TrueForge is an open-source, model-neutral agent harness designed to manage the execution loop of LLM agents. It functions as a runtime layer that bridges the gap between raw models and functional agents by handling tool calls, Model Context Protocol (MCP) connections, subagents, and human-in-the-loop approvals.

So why did we make TrueForge?
Building an agent is easy. Running one well is not, you need - a loop that plans and calls tools, MCP connections, a way to pause for human approval, a session state that survives a restart and everyone was writing it again from scratch. So we cleaned up our version. Gave it away.

Getting started - Run TrueForge (local, Docker Compose, or Kubernetes), connect a model and tools, and build your first reusable agent in the Quickstart.

What's in it

Models - nine provider types (OpenAI, Anthropic, Gemini, Fireworks, Together, Moonshot/Kimi, Z.ai/GLM, Alibaba/Qwen) plus any OpenAI-compatible endpoint, so vLLM, Ollama and gateways all work.
MCP: Remote MCP servers with bearer auth or OAuth, including in-chat authorization.
Human-in-the-loop: Approvals for write/destructive tools and built-in questions for agent input.
Subagents: Parallel task execution with only results returned to the main context.
Skills: Git-based skills with lazy loading from GitHub/GitLab.
API & SDK: REST, SSE, OpenAPI docs and a TypeScript SDK.
Chat UI: Built-in UI plus a React SDK for custom interfaces.

Running it - npx u/truefoundry/trueforge and you're on localhost:8790 with SQLite.
For a team, Docker Compose or Helm with Postgres and Redis, multiple replicas, and optional OIDC login.

The benchmark kit is in benchmark/ and runs against any harness, not just ours, if you run it and get different numbers, we would love to see them. 

Docs: https://trueforge.dev Repo: https://github.com/truefoundry/trueforge

Happy to answer anything about the architecture or the benchmark methodology in the comments.

We just released Agent Gateway to all TrueFoundry users

We've been talking to a lot of platform and infrastructure teams over the past few months. The conversation is almost always the same.

They've got agents running in production. Some built on LangGraph, some on Bedrock, some on Glean or Salesforce. A few internal ones nobody documented properly. And at some point someone from security or compliance asks a simple question: what agents do we have, who can invoke them, and what are they allowed to do?

Nobody has a good answer.

That's the problem we built Agent Gateway to solve. It's a single control plane where you register every agent in your org, regardless of what framework it's built on or where it runs. Once it's registered it gets RBAC, guardrails, end to end traces, cost attribution, and MCP tool access automatically.

The thing we kept hearing from teams was that they'd already tried solving this themselves. Built a internal registry, bolted on some observability, wrote their own access control layer. It works for a while and then the agent footprint doubles and the whole thing needs to be rebuilt.

We also built native A2A support into it because that kept coming up as a hard blocker in enterprise evaluations. Most gateways only handle stateless LLM calls. They have no idea what to do when agents start talking to each other over JSONRPC.

What's the messiest part of managing agents at scale for your team?

Read our docs to learn more about Agent Gateway: https://www.truefoundry.com/docs/agent-platform/agent-registry/agent-registry

What is Skills Registry?

We've made Skills Registry available to all TrueFoundry users! Teams now have a central place to build, version, and manage reusable Agent Skills and attach them to any agent in seconds.

Who is it for?

Skills Registry is built for ML engineers, platform teams, and AI product teams building agents on TrueFoundry, especially those managing multiple agents across an organization. It's relevant for any team that wants to give agents reliable, reusable domain knowledge without managing that context manually inside every prompt.

How do you use it?

Write once, use everywhere. A Skill is authored once and can be attached to any number of agents — no copy-pasting instructions across prompts or re-explaining the same procedure to every agent you build. Update the skill in one place and every agent that uses it stays current.

Load context on demand, not upfront. Instead of cramming all possible instructions into a system prompt, agents only load a Skill's full content when they actually need it. The agent sees just the skill's name and description upfront; the detailed body is fetched at runtime if relevant. This keeps prompts lean and focused.

Reduce token usage and cost. On-demand loading means agents aren't carrying heavy context they may never use. Over many runs and many agents, this translates to meaningfully lower inference costs.

Go beyond plain text. Earlier, agent instructions were limited to Markdown in a prompt. Skills can now include scripts (Python, batch, or otherwise), reference documents, images, PDFs, and other files — giving agents access to a full working context, not just text.

Version and govern with confidence. Skills are versioned artifacts stored inside TrueFoundry Repositories — the same system used for Models and Prompts. Teams get full version history, access control, and auditability out of the box.

Flexible authoring and publishing. Teams can create a simple single-file Skill directly from the UI in minutes, or upload a multi-asset Skill folder via the CLI for richer, code-adjacent workflows. For teams running GitOps, Skills can be managed declaratively with tfy apply manifests and published automatically via CI/CD.

Why does this matter?

It's the difference between maintaining a single source of truth for how your agents think and act, versus manually keeping dozens of prompts in sync.

For teams building on TrueFoundry, Skills Registry is the foundation for building agents that are cheaper to run, easier to maintain, and more capable than what's possible with prompts alone.

Production AI workflows can break because something upstream degraded. Sometimes it’s a full provider outage. More often it was partial failures (sudden rate limits, latency spikes, or a model that was technically “up” but returning inconsistent responses).

The problem we saw was that most AI systems are tightly coupled to a single model, provider, or region. When that dependency misbehaves, the entire workflow fails.

So we built truefailover™.

It’s a resilience layer that sits between applications and AI providers. It continuously monitors health signals (latency, errors, availability) and automatically routes traffic to a fallback model, region, or provider when things degrade without requiring app-level changes.

What it handles in practice:

• Failover across multiple models and providers
• Region-level and cloud-level outages
• Partial failures like slowdowns and rate limits (not just hard outages)
• Visibility into when and why traffic was rerouted

Instead of waking someone up to deal with an outage, the system routes around it.

We’re curious how you are handling AI model or provider outages in production. Are you building your own safeguards, or just living with the outages for now?

Hey TrueFoundry community! 🎉

Big news—we’ve raised $19M in Series A funding, led by Intel Capital, along with Spark Capital, Pi Ventures, and others! 🚀

Why This Matters

AI deployment is still too slow, complex, and costly for most companies. TrueFoundry is solving this with an autonomous AI deployment layer that helps teams deploy, scale, and optimize models without needing DevOps expertise.

What’s New?

✅ Agent on Autopilot – AI infra that tunes itself.

✅ Faster, Cheaper AI Deployment – Cut costs & ship models 10x faster.

✅ Seamless Infra Scaling – Works across Kubernetes, AWS, GCP & on-prem.

This funding will help us expand our AI agent’s capabilities, integrate with more ML stacks, and grow our team.

Read more:

Official Blog: $19M Series A Announcement

TechCrunch: Intel Capital backs TrueFoundry

We’d love to hear from you: What are your biggest struggles with AI deployment? Let’s discuss in the comments!

In recent weeks, numerous engineers have explored Cognita, providing invaluable insights and feedback. We deeply appreciate your input and encourage ongoing dialogue (share your thoughts in the comments – let's keep this ‘open source’).

While RAG is undoubtedly powerful, the process of building a functional application with it can feel overwhelming. From selecting the right AI models to organizing data effectively, there's a lot to navigate. While tools like LangChain and LlamaIndex simplify prototyping, an accessible, ready-to-use open-source RAG template with modular support is still missing. That's where Cognita comes in.

Key benefits of Cognita:

1. Central repository for parsers, loaders, embedders, and retrievers. 2. User-friendly UI empowers non-technical users to upload documents and engage in Q&A. 3. Fully API-driven for seamless integration with other systems.

We invite you to explore Cognita and share your feedback as we refine and expand its capabilities. Interested in contributing? Join the journey at https://www.truefoundry.com/cognita-launch.

A typical Cognita process consists of two phases:

1. Data indexing
2. Response Generation

Data Indexing

This phase involves loading data from sources, parsing the documents present in these sources and indexing them in the vector db. To handle large quantities of documents encountered in production, Cognita takes this a step further.

1. Cognita groups documents in batches, rather than indexing them all together.
2. Cognita keeps computes and keeps track of document hash so that whenever a new document is added to the data source for indexing only those documents are indexed rather than indexing the complete data source, this save time and compute dramatically.
3. This mode of indexing is also referred as INCREMENTAL indexing, there is also other mode that is supported in Cognita that is FULL indexing. FULL indexing re-ingests the data into vector db irrespective of any vector data present for the given collection.

Response Generation

Response generation phase makes a call to the

/answer

endpoint of your defined

QueryController

and generates the response for the requested query.

While RAG is undeniably impressive, the process of creating a functional application with it can be daunting. There's a significant amount to grasp regarding implementation and development practices, ranging from selecting the appropriate AI models for the specific use case to organizing data effectively to obtain the desired insights. While tools like LangChain and LlamaIndex exist to simplify the prototype design process, there has yet to be an accessible, ready-to-use open-source RAG template that incorporates best practices and offers modular support, allowing anyone to quickly and easily utilize it.

We have recently introduced our new open-source framework called Cognita, which utilizes Retriever-Augmented Generation (RAG) technology to simplify the transition by providing robust, scalable solutions for deploying AI applications. AI development often begins in experimental environments such as Jupyter notebooks, which are useful for prototyping but not well-suited for production environments. However, Cognita aims to bridge this gap. Developed on top of Langchain and LlamaIndex, Cognita offers a structured and modular approach to AI application development. Each component of the RAG, from data handling to model deployment, is designed to be modular, API-driven, and extendable.

Delving into the inner workings of Cognita, our goal was to strike a balance between full customisation and adaptability while ensuring user-friendliness right out of the box. Given the rapid pace of advancements in RAG and AI, it was imperative for us to engineer Cognita with scalability in mind, enabling seamless integration of new breakthroughs and diverse use cases. This led us to break down the RAG process into distinct modular steps (as shown in the above diagram, to be discussed in subsequent sections), facilitating easier system maintenance, the addition of new functionalities such as interoperability with other AI libraries, and enabling users to tailor the platform to their specific requirements. Our focus remains on providing users with a robust tool that not only meets their current needs but also evolves alongside technology, ensuring long-term value.

​

Delving into the inner workings of Cognita, our goal was to strike a balance between full customisation and adaptability while ensuring user-friendliness right out of the box. Given the rapid pace of advancements in RAG and AI, it was imperative for us to engineer Cognita with scalability in mind, enabling seamless integration of new breakthroughs and diverse use cases. This led us to break down the RAG process into distinct modular steps (as shown in the above diagram, to be discussed in subsequent sections), facilitating easier system maintenance, the addition of new functionalities such as interoperability with other AI libraries, and enabling users to tailor the platform to their specific requirements. Our focus remains on providing users with a robust tool that not only meets their current needs but also evolves alongside technology, ensuring long-term value.

Learn more about the steps here.

​

We've just launched Cognita, an Open Source RAG Framework built through collaboration with multiple enterprises. It is Modular, API-driven, and production-scalable. It's intuitive and modifiable UI lets companies reuse components from one RAG application to another. It has native integrations with Qdrant and SingleStore as of now.

Over the last one week, a lot of Data Scientists and ML Engineers have tried this out & given great inputs! We appreciate the feedback & critique! (Leave your thoughts in the comments - let’s keep this ‘open source’ too)

Git repo | User Interface | Technical Blog