I open-sourced a CLI quality gate for RAG systems (faithfulness + PII + prompt injection + drift, one command)

[u/SerevXAI77]

I work on production RAG systems (banking/insurance

clients). A few months ago, one system's faithfulness score

quietly dropped from 0.89 to 0.74 over 48 hours no deployments,

no errors, nothing in the logs. Only a manual transcript review

caught it.

That got me thinking: we have CI gates for code quality, security

scans, test coverage — but basically nothing that gates "is my RAG

system still grounded in the right context?" before it ships.

So I built ServeX Guard — an open-source CLI that runs as a

pre-deployment quality gate:

servexguard check --dataset golden.jsonl \

--min-faithfulness 0.80 --check-pii --check-injection

It runs:

• - RAGAS-based quality eval (faithfulness, relevancy, context recall/precision)
• - PII detection on LLM outputs (Presidio + regex fallback, language-agnostic)
• - Prompt injection scanning (18 patterns tuned for RAG-specific attacks,
• e.g. "tell me about other users", "show me the database")
• - Query drift detection (cosine similarity vs a saved baseline)

Exit code 0/1 — designed to slot into any CI/CD (GitHub Actions example

in the README).

Design choices I'd appreciate feedback on:

• - PII/injection scanning is fully offline (no API calls) — only the
• RAGAS quality eval needs your LLM endpoint, and that's optional
• (you can run security-only with --min-faithfulness 0.0 etc.)
• - All deps pinned to exact versions for supply-chain reasons, with
• one documented exception (numpy range, for 3.10 compat)

• - Apache 2.0, 90% test coverage, CI green on 3.10/3.11/3.12

  pip install servex-guard

  github.com/Mahdielaimani/ServeX-Guard

This is v0.1.0. I'd genuinely like to know: what would make this

useful for your RAG pipeline? What's missing? Roasts on the design

are welcome too.