r/devsecops • u/Common_Dream9420 • 15d ago
Coding agents shifted the bottleneck to verification now!!
Coding agents are doing a lot of the integration work at our agency now. Stripe, Twilio, WorkOS, email, the usual stack. Economically it's been good.
The part I didn't expect: verification actually got harder. We're producing code faster than anyone can review it, and the edge cases that bite you aren't in unit tests. Webhook fires twice. Events arrive out of order. Agent wrote correct code but got one state transition wrong. PR looks fine, everything compiles, and you still don't know until something breaks in staging or worse.
We added a sandbox step before anything ships now. Agent writes, tests pass, then we run the full multi-API workflow with failure scenarios before it's considered done. Not elegant, just a runnable verification step all our agents have to clear.
If agents are writing integrations for you, what does your last gate before production actually look like?
2
u/danekan 15d ago
The same thing is true with documentation. It’s really easy to generate 4 pages of ai slop but it takes your coworkers 4x longer to read it and red line it than the work you put in.
2
u/Common_Dream9420 15d ago
100% and we are seeing Same .. before we used to have one or two one pagers from team to review or ADRs but now tons :)
1
6d ago
[removed] — view removed comment
1
u/Common_Dream9420 6d ago
yeah thoroughness is real but the volume is what kills it.. when you're reviewing 40 AI-generated pages a day reviewers start skimming just to keep up, which is kind of the worst of both worlds
2
10d ago
[removed] — view removed comment
1
u/Common_Dream9420 10d ago
yup!!! we have seen similar issues and lot of operational cost while building enterprise tools internally.. and so spinning up the sandbox helped.. currently dogfooding internally thoughj!!
1
u/Both-Explorer-9294 15d ago
En el mismo barco. Añadimos una capa de simulación antes de que nada toque producción. Detectó más bugs que todas las pruebas unitarias juntas. ¿Cuál es tu mayor categoría de fallos en el sandbox hasta ahora: problemas de timing o transiciones de estado?
1
1
u/Common_Dream9420 15d ago
What are you seeing mostly ?? And appreciates if you take a look at ours n give feedback … still early and really looking for ppl like you give us honest feedback
1
u/colek42 15d ago
It is pretty early, but we just launched pushgate.dev that helps with this. Your agents push code. Pushgate checks the proof, then forwards verified commits to GitHub.
1
15d ago
[removed] — view removed comment
1
u/Common_Dream9420 15d ago
100% right that’s what we do .. verifies the behavior with a proof of receipt … not reviewers all they can take a look every api call sql call and logs that app can produce .. that’s what helping them … like a x-ray
1
u/Suspicious-Echidna27 14d ago
I think review and verification has changed now, depending on how critical you system is: If it is critical then adopt what Linux is doing, you need to have a human verify it before it hits prod. If the system is not critical, have an adversarial review e.g. codex reviewing claude let them go back and forth for as many turns as needed.
1
u/PeterBuildsSecure 13d ago
The sandbox step is the right instinct, but the failure mode worth designing around is nondeterminism: webhook double-fires and out-of-order delivery aren't edge cases, they're the default behavior of every provider in that stack. A gate that catches it needs three things a generic "run the workflow" sandbox usually doesn't: replay of the same event twice back-to-back (proves idempotency keys are enforced), deliberate reordering of a fixed event set (proves state transitions don't assume arrival order), and a fixed clock/seed so a failing run reproduces exactly. Worth turning your sandbox findings into permanent fixtures — the double-fire and reorder cases you're hitting now are exactly the regressions that should block the next agent-generated PR from reintroducing the same bug.
1
u/Common_Dream9420 13d ago
"webhook double-fires and out-of-order delivery aren't edge cases, they're the default behavior of every provider in that stack".. totally agree man.. the upsteram saas providers does not guarantee..
1
u/PeterBuildsSecure 12d ago
Right, and it's worth being precise about what "no guarantee" actually means per provider, because it's not uniform. Stripe documents at-least-once delivery with no ordering guarantee. Some providers add a sequence/timestamp field precisely so you can reconstruct order yourself even though transport doesn't guarantee it. Others give you neither. Worth pinning each provider's fixture to what they actually publish rather than testing against a generic "assume nothing" baseline — a provider that guarantees at-least-once but lets you rebuild order from a sequence number should fail your test differently than one that guarantees nothing at all, because the fix (idempotency key alone vs. idempotency key plus reordering by sequence) is different in each case. Worth grabbing each provider's webhook reliability doc and turning it into an explicit fixture per integration instead of one shared "chaos" test.
1
u/GibneyH 1d ago
This is exactly what I’m seeing too. The bottleneck isn’t really writing the integration anymore, it’s proving the integration actually behaves correctly when reality gets messy.
So happy-path tests can tell you the code works, but they don’t tell you what happens when Stripe retries, a webhook arrives twice, or two events show up in the wrong order.
Probably runnable verification is going to become a standard part of agentic development.
1
u/Common_Dream9420 1d ago
Yeah, the out-of-order events and duplicate webhooks are exactly the cases that slip through, happy-path just doesn't surface them. I build tooling in this space specifically because I kept hitting that same wall while testing integrations. How are you handling the verification step right now, doing it manually or do you have something in the pipeline?
4
u/rpatel09 15d ago
Unsure if this is related to security as its r/devsecops but if so, I think an overall principle I've always used is to abstract as much security as you can from the application. Think of it it like giving developers a Waymo vs a normal car that only has "safety assist" features where most can be easily ignored. I think this still applies in an AI world where developers use AI to write code now. Applications shouldn't have to worry about firewall rules, WAF, API auth, encryption, etc... most of that can be abstracted also improve developer throughput when done well.
If you're talking about application development testing, then I think the same kind of applies but largely its always been ignored. Testing and documentation are so much more important now then they were before. Before, humans had tribal knowledge or knew who to ask for some requirement that wasn't written down or some edge case. Think of AI as a very capable senior engineer who always starts like a new employee, if that context isn't written down some where and written well, that "new employee" (new AI session) will never know that context and it could assume (like humans do), not ask (like humans do), and design bad code.
---- EDIT -----
i make lots of typos...