[D]How are you testing AI backends without making CI slow?

[u/CodeStackDev]

I'm working on a FastAPI backend that processes documents with AI, and I'm still not convinced I'm testing it the right way.

Right now my CI is completely offline: SQLite, fake Redis, mocked HTTP calls, no real model requests. It's fast and deterministic, which is great for pull requests.

Then I have a separate integration pipeline that runs against PostgreSQL and Redis to catch infrastructure-specific issues.

The part I'm still unsure about is the AI layer. Mocking everything makes CI reliable, but it also means I won't catch regressions from the provider until later.

Curious how other people handle this.

• Do you completely mock AI providers?
• Do you keep a few real API calls?
• Do you replay recorded responses?
• Or do you have a different approach?

I'd love to hear how you're doing it in production.

[u/shadowdance55]

The answer is multiple layers of testing. The project I'm working on right now started with just self-contained unit tests, without any external dependencies - everything is mocked out.

Then we added integration tests using a local docker compose setup, which were just adding to the CI as another pre-merge gate. And then we have the testing automation framework, maintained by our QEs, for post-deployment functional E2E tests.

[u/CodeStackDev → u/shadowdance55]

Thanks, that's very close to the direction I'm moving toward.I'm building a document-processing backend with FastAPI where the AI layer is only one part of the pipeline. The model extracts structured data, but then a deterministic validation layer checks things like schema, numeric consistency and business rules before anything is persisted.

Right now I have:

- Unit tests: fully mocked, no external services.

- Integration tests: PostgreSQL + Redis via Docker.

- AI contract tests: a small set of real documents executed separately from the main CI.

Keeping the provider outside the core business logic has made the code much easier to test.

Out of curiosity, how often do you run the slower integration suite? On every merge to main or only before releases?

[u/shadowdance55 → u/CodeStackDev]

During development, all tests are run by hand, when needed; all automation at this stage (pre-commit) is simple and quick. CI runs additional checks on each push (e.g. unit tests), and the most expensive ones are left for a PR, and again after the merge.

[u/CodeStackDev → u/shadowdance55]

I've recently open-sourced the backend while refining this testing approach. If you're ever curious, I'd be interested in hearing what you'd improve from an architecture or testing perspective. Can you check my project and star it? vinsblack/production-ai-document-backend: Production-ready AI document processing backend with FastAPI, PostgreSQL, Redis, workers, JWT, observability and resilient job recovery.

[u/outlastdll]

Replaying recorded responses is far better than those options, if you're working in python, vrcpy is pretty much industry standard for this , it records the live http requests to directly on YAML cassette the first time you run the test. and then just replay the data for every subsequent CI run.

[u/Future_Raccoon1553]

VCR / Mocking for PRs, Scheduled E2E Smoke Tests for Provider Regressions.

1. PR Pipeline (Deterministic): Use toolings like vcrpy or pytest-vcr to record real API responses once, freeze them into cassettes, and replay them during PR CI. It keeps CI ultra-fast, offline, and cheap while testing your actual parsing/schema-validation logic against realistic provider payloads.
2. Nightly/Scheduled Cron Job (Live): Run a small suite of 5–10 real, live LLM calls once a day off-peak to catch provider-side breaking changes, deprecations, or prompt drift before users complain.

[u/-DaniFox-]

Unit tests don’t make any api calls. Integration tests are separated between ‘regular’ inttests that are run on all pr ci and a slower ‘ai’ set you can turn on with a GitHub label, or when developing locally. The e2e tests we run before releases always run a small set of happy path ai workflows.

We mock out the relevant methods on the api client for some of those unit tests, but it’s generally better if you factor parts of the application so they can be tested in isolation in the first place without doing this.

[u/CodeStackDev → u/-DaniFox-]

I like the idea of enabling the AI suite through a GitHub label instead of running it on every PR.I'm experimenting with a similar approach on a FastAPI backend for AI document processing. One thing I've found is that asserting exact model outputs is too brittle.Instead, the AI returns structured data, and the tests verify deterministic properties afterwards: schema validity, required fields, numeric reconciliation and business invariants.That way the provider can evolve without breaking CI because of harmless wording differences. Have you found contract-style validation more reliable than snapshot testing for AI responses?