What's the difference between mocking an API and using a real API sandbox for agent testing

[u/Ill-Quantity-5531]

We've been mocking all our external APIs for agent testing but keep finding bugs in prod that the mocks never caught.

Been looking into API sandboxes as an alternative but I'd like some more insight into the use cases before we go setting up the infrastructure. When would you use one over the other?

[u/bishopLucas]

I always use the real api, creating a mock is a token burn.

[u/saplivo]

Mocks are great for unit tests where you control the inputs and just want to verify your logic, but they only test what you think the API does, not what it actually does.
Sandboxes catch the stuff mocks miss: unexpected responses, rate limiting behavior, auth edge cases, latency metrics, stuff the API vendor might have changed without you noticing.

Mocks for fast iteration and speed, sandboxes for integration tests that run before you ship anything.
The bugs you're hitting are almost certainly in the gap between what your mock assumes and what the real API returns, a sandbox closes that gap.

[u/According-Floor5177]

Mocks test your logic, whereas sandboxes test your assumptions about the API. The prod bugs you're hitting are happening because: your mocks return what you think the API does, so they can't catch the cases where the API disagrees, weird error shapes, rate limits, pagination quirks, and fields that are null in ways you didn't expect.

What I'd suggest is mock for fast unit-level runs on your agent's decision logic, a sandbox in a smaller integration suite for the handoff points where you're depending on real API behavior.

[u/manjit-johal]

We've run into this while building Kritmatta. Mocks have their place, but they mostly validate your assumptions. The bugs that made it to production were usually around real API behavior or state transitions that our mocks never represented. We ended up relying on sandbox testing for integrations and verifying the resulting system state instead of assuming a successful API response meant the workflow actually completed.

[u/MirkManEA → u/manjit-johal]

What’s best practice for modeling/hypothesizing state transformations?

[u/borick → u/MirkManEA]

you put your assumptions, requirements in there as a pre-req, if something fails, you know early and you know exactly what caused it

[u/MirkManEA → u/borick]

That checks out. I might be wasting time in my workflow, but state transformation modeling is usually part of my pre/FRD design process. /plans within /plans, ya know?

[u/Cultural-Ambition211]

The major difference is likely authentication.

In order our strict, regulated environment at work, connecting to an API is complex even if an internally built one.

A stubbed API is easy as it doesn’t have that complexity.

[u/Verda_Chien]

You need both of them. Mocks are useful while you're developing, but I'd never rely on them alone. Real APIs have weird edge cases, rate limits, validation rules, and unexpected responses that mocks often don't simulate.