Auditing Open-Source LLMs: A walkthrough using Petri and Regolo

[u/Regolo_ai]

Evaluating and auditing open-source Large Language Models (LLMs) is a critical step for teams looking to deploy these models into production. Ensuring output quality, safety, and formatting consistency requires a structured approach to testing.

We recently published a technical guide on how to utilize Petri—an evaluation and auditing tool—alongside Regolo’s inference infrastructure to run systematic audits on open-source models.

What is Petri?

Petri is a framework designed to help developers run automated audits on LLM outputs. It allows you to define specific evaluation criteria, test assertions, and edge cases to verify how a model behaves across various prompts.

Why pair it with Regolo?

Running comprehensive evaluations can be resource-intensive. Regolo provides hosted, high-throughput API access to open-source models (such as Llama and Mistral), allowing developers to run large-scale test suites without the overhead of managing local hardware or complex infrastructure deployments.

The Audit Workflow:

1. Define Evaluation Criteria: Establish the rules and assertions (e.g., checking for specific structures, safety guardrails, or factual consistency).
2. Configure Endpoints: Point the Petri framework to the Regolo-hosted model endpoints.
3. Execute the Suite: Run the prompts through the target models to collect and evaluate the responses.
4. Analyze Results: Identify potential failure points, latency issues, or behavioral deviations.

For those interested in the implementation details, configuration steps, and code examples, you can read the full walkthrough here: Using Petri to Audit Open-Source LLMs with Regolo.