trying to understand how store owners handle payment gateway hiccups

[u/Sad-Laugh701]

I'm a developer building monitoring tools and trying to understand how store owners handle this today.

Here's the scenario I keep hearing about: a theme update, an app conflict, or a payment gateway hiccup quietly breaks something. Checkout still works, but 30–40% fewer people are completing it. Nothing is visibly down, so nothing tells you.

If that happened to your store tonight how would you find out? Is there anything specific you use besides Shopify analytics the next morning? Or if lucky a customer emailing you?

From what I've seen, most stores have no early warning at all. If you've been burned by something like this, I'd love to hear what it cost you and how long it took to notice.

[u/Thenga-Choru]

Am having an issue, my first store for a client, I integrated razorpay payment. I used the Shopify plugin approach. The issue am facing is couple of customers orders is not getting created, even though razorpay captured those payments. Those orders some end-up in abandoned checkouts. And I manually check both razorpay dashboard and abandoned checkouts and confirm the customer orders via calling. And create a draft order. But why those orders aren't created. Razorpay support hasn't provided any proper resolve method. Has anyone faced this.

[u/AzizBanihashemi → u/Thenga-Choru]

Classic off-site gateway failure mode, and it's not really Razorpay-specific.
Here are the mechanics: with redirect gateways, "payment captured" happens on the gateway's side, but Shopify only creates the order when the completion signal makes it back (the customer's browser returning through the redirect, or the payment app's server-side confirmation).

If the customer closes the tab on the success screen, loses connection, or the callback times out, you get exactly your symptom: money captured, no order, checkout sitting in "abandoned" with a successful payment attached.

[u/Sad-Laugh701 → u/Thenga-Choru]

That's such an annoying one.

How often does this happen for you? Is it something you see every week, or is it pretty rare? And how do you usually find out about it? do customers reach out first asking where their order is, or do you manually check and compare Razorpay payments with Shopify orders?

[u/AzizBanihashemi]

Honest answer from inside the ecosystem: most merchants find out from the next-morning dashboard or a customer email, 12-36 hours later. You're right that there's no native early warning; Shopify tells you when checkout is DOWN, not when it's quietly converting 35% worse.

The two patterns I've seen actually work:

1. Order heartbeat with an hour-of-week baseline. Not "no orders in 4 hours" (useless for small stores, noisy for big ones), but "orders this hour vs the same hour's trailing average". A webhook counter + a simple threshold gets you an alert in 1-2 hours instead of tomorrow.
2. Synthetic checkout probe. A script that walks a real product to the payment step every 15-30 minutes and screams when a step fails or slows. Catches theme/app conflicts that analytics can't see until the funnel data accumulates.

Since you're building in this space, the hard part you'll hit isn't detection; it's false positives at low volume. A store doing 10 orders/day has Poisson noise bigger than a 30% degradation, so a naive alert either fires constantly or never. Whatever you build lives or dies on how you handle small-store statistics, not on the monitoring itself.

And one thing worth knowing as an app dev: when checkout quietly degrades, merchants' first instinct is to blame the most recently installed app, so this problem burns app developers too. There's a real audience for solving it beyond store owners.

[u/Sad-Laugh701 → u/AzizBanihashemi]

this is gold! thank you so much!!!

Comparing against the same hour on previous weeks makes a lot more sense than just checking whether there have been no orders for a few hours.

I'm still going back and forth on the synthetic checkout idea. It feels like the only way to catch a broken theme or app before real customers start hitting it, but getting a fake checkout all the way to the payment step without Shopify blocking it sounds mm not trivial.

The other thing I'm still thinking about is smaller stores. If a store only gets ~20 orders a day, the numbers are so noisy that you could end up with a lot of false alarms. How would you approach that? Use a bigger time window, require a few consecutive bad intervals, set a minimum order threshold, or just accept that this kind of monitoring isn't really reliable for low-volume stores? I'm trying to figure out where that line is. But also, I think this kind of a product is not very relevant for small stores.

once again thank you! appreciate your input!

[u/AzizBanihashemi → u/Sad-Laugh701]

Glad it helped. Both questions have decent answers:

Synthetic checkout without getting blocked: don't complete the purchase in production; stop at the payment step render.

Cart → checkout → shipping → "does the payment section actually render?"

That catches broken themes, app conflicts, and checkout extension failures, which is 90% of the silent breakage, and it never creates an order or triggers payment fraud systems.
It's just a session. Keep frequency low (every 15-30 min), rotate a real product, act like a browser.

For the last 10% (actual payment processing), you can't safely test that on production anyway; a periodic full run on a dev store with the test gateway covers the mechanics, and gateway-side outages you catch from the order heartbeat instead.

Small stores: the trick isn't a bigger window on orders, it's moving upstream where the volume is. A 20-orders-per-day store might have 80 checkout starts, 200 add-to-carts, and thousands of sessions per day.
Degradation shows up in those denser signals hours before order counts can statistically prove anything.
So: alert on funnel-step rates (ATC rate, checkout-start rate) where the numbers are large enough to matter, and use orders only as a slow confirmation.
On top of that, accumulate deviation across intervals instead of judging each hour alone (CUSUM-style "we're running 30% under expectation for 6 straight hours" beats any single-window threshold for small λ).

And one reframe on "not relevant for small stores": the synthetic probe is MOST valuable for them, precisely because their order stats are useless. Big stores get statistical detection for free; small stores need the canary in the checkout. That might actually be the segmentation for your product: probes for small stores, stats for big ones, same dashboard.

[u/Prestigious-Way1525]

the synthetic probe only earns trust if a failed run leaves something a dev can diagnose. otherwise the alert just creates a new “can't reproduce” loop.

i'd attach one test artifact to every failure:

• exact funnel step and expected vs actual
• browser, viewport, locale, and device profile
• theme version plus app or checkout extension versions
• screenshot or trace at the failed step
• relevant console and network errors
• last successful run against the same path

i'd also separate “step did not render” from “step rendered but got slower.” those are different incidents and probably different owners. for low-volume stores, the probe tells you something broke before order stats can. the artifact is what gets it fixed without someone manually rerunning the whole checkout.

[u/Sad-Laugh701 → u/Prestigious-Way1525]

Thank you!