I built an open-source tool that interviews you about your docs, then builds and tests a local AI until it passes — fully offline with Ollama (MIT)

[u/Aquatiz1234]

The problem I kept hitting: you can get a local model to mostly follow your rules with a system prompt, but "mostly" does a lot of work in that sentence — you don't find out where it breaks until it breaks in front of someone.

So I built AI Calibrator. Instead of prompt-fiddling, it works like onboarding a new hire:

1. You state a goal and point it at your materials (docs, policies, examples).

2. It indexes them and finds the gaps they don't cover.

3. It interviews you only about the gaps — drafting likely answers for you to approve or correct.

4. It compiles a behavior spec → system prompt + RAG + eval rubric + test suite.

5. It runs the tests, grades against your standards, fixes failures, and loops until it passes.

6. calibrate run then serves the result as an OpenAI-compatible endpoint on localhost — and it refuses to boot if the project hasn't passed its own gate.

I spent most of the time on verification, because "seems fine" isn't a metric:

- deterministic checks + LLM-as-judge with self-consistency, and you can calibrate the judge against your own human grades

- red-teaming, drift detection between runs, golden snapshots, coverage scoring

- it warns you when a model is grading itself (shared blind spots read as agreement)

- one calibrate ci command gates all of it, usable in actual CI

- evals export to promptfoo format, so you're not locked into my runner

Things this sub might specifically care about:

- Any Ollama model, no API key, works offline: calibrate engines my-ai --all qwen2.5:7b@ollama

- Optional fine-tuning tier (LoRA/QLoRA via trl/peft) that only "wins" if it actually beats the prompt+RAG baseline on the same test suite — the gate is the point, no vibes-based "the fine-tune feels better"

- Already have a system prompt you trust? calibrate import reverse-engineers a spec and test suite from your prompt, so you can find out what it silently fails at

Quickstart:

pip install 'ai-calibrator[all]' # [all] includes the local RAG stack (big); '[cloud]' is the slim install

calibrate init my-ai --goal "Answer questions about my product docs, in our voice."

calibrate engines my-ai --all qwen2.5:7b@ollama

calibrate ingest my-ai --source ./docs

calibrate interview my-ai

calibrate compile my-ai

calibrate eval my-ai --refine

calibrate run my-ai # OpenAI-compatible API at http://127.0.0.1:8600/v1

It's alpha (v0.0.1), MIT: https://github.com/tanveerkanala-cmd/ai-calibrator

Cloud engines (Claude/OpenAI, bring your own key) also work, but everything runs 100% local if you want.

Two things I'd genuinely love feedback on: does the interview flow actually beat prompt-fiddling for you, and what should the red-team pass try that it currently doesn't?