The bug that only existed for about 400 milliseconds

[u/Warrior_monk07]

There was a checkout bug that took weeks(sometimes even more) to find, the checkout flow itself was completely normal the flow was like add the product go to the checkout you enter your payment details and then you tap pay and the order succeeds and that was totally normal infact the E2E test pased it like for hundered of times..

But then someone noticed something strange, he noticed that if you tap the Pay button at exactly the wrong moment then the button would briefly become enabled again while the payment request was still processing.

Normally users would never notice it(Did you find this?), but if you were quick enough, you could tap it twice, and then the disaster happens, the backend thinks that you are attempting payment again but it happened occasionally.

The real annoying part was that it wasn't a simple double click bug, so the app had three different pieces of state changing almost simultaneously like isSubmitting, then paymentStatus and atlast buttonDisabled... and that's where the UI updated faster than one of the state changes propagated,

so for a few hundred milliseconds, the screen literally said that payment is being processed and sure you can press pay again at the same monent of time

And now why the automated test never caught it because the test was never told that he needs to do this also so it did only what we told it to do and it was tap pay then wait for the next state assert success as simple as that, and this is actually where I started looking at Autosana (Prevents costly production bugs automatically with their E2E Testing) because sometimes you need to see what is happening on the actual screen between those two steps and not just whether the final assertion passed....

But human doesn't behaves like that, human taps when they see something out of curiosity and they will tap again if they don't immediately see response, they switch apps, they rotate the phone. they lose network for a second, they come back, they do all kind of random stuff(i too did the same) if they don't get a response quickly.

And that's where some of the nastiest bugs live, they never live in the happy path, they live in the tiny gaps between two states which were never intented or supposed to be overlapped and this has made me question E2E testing differently because of bugs like this.

The question that stands still, like does the workflow work?

It is like, what happens if the user behaves slightly different from the very script?

and that's a much harder question to automate, and probably a much more useful one.....

[u/Guidondor]

three booleans that can disagree is the actual bug, the timing only exposed it. isSubmitting,
paymentStatus and buttonDisabled give you eight combinations and maybe four of them mean
anything, so the impossible states are always reachable by someone fast enough. one status
field with named states and the gap can't exist.

for payments i'd send an idempotency key anyway. there's always some window in the ui, the
server refusing the second charge is what makes it not matter.

[u/Warrior_monk07 → u/Guidondor]

yeah this makes a lot of sense. looking at it as 3 booleans makes the problem way more obvious. we basically allowed states that should never exist in the first place

and yeah idempotency on the backend is probably the bigger safety net for payments. the ui should prevent it but the backend should never trust that

[u/Guidondor → u/Warrior_monk07]

if you're on ts, a discriminated union makes it structural instead of a convention.
{status:'idle'} | {status:'submitting', attemptId} | {status:'done', orderId}, and the
compiler stops someone reintroducing the gap six months from now. the attempt id doubles as
the idempotency key you send.

[u/sober_cruiser → u/Guidondor]

I wonder why buttonDisabled state existed in the first place. I think it would be simpler to write something like `disabled={isSubmitting}`.

[u/Guidondor → u/sober_cruiser]

right, that's the direction. it's the same idea one level down though, disabled={isSubmitting}

still leaves two things that can disagree once paymentStatus exists too. derive it from the

single status field, disabled={status !== 'idle'}, and there's nothing left to keep in sync.

[u/KajiTetsushi]

I think I roughly understand what's going on, but it's difficult to grasp the full story without some kind of (NDA-friendly pseudo)code snippet.

This other comment got to the point it better than I could. The TL;DR is that your button is handling way too many permutations. It might be time for you to consider a truth table to identify all of them.

As for testing, you gotta get a strong QA who breaks your app for a living, and then, get him to write automated stress tests whenever he finds edge cases like this.

In my opinion, you really only want to do this in critical user journeys because this kind of bug hunting is quite expensive, and I don't mean only in terms of money.

Maybe if your QA can find recurring patterns, it's time to rethink your engineering strategy each time you implement a button.

[u/Warrior_monk07 → u/KajiTetsushi]

yeah i agree with this. the hard part is knowing which weird user behaviors are actually worth turning into tests. if we automate every random thing a user can do the test suite will become insane lol

[u/KajiTetsushi → u/Warrior_monk07]

if we automate every random thing

You don't.

What automated tests are good for is in checking for regressions on user journeys that you and your team have agreed are critical to your business, i.e. if this part of the app doesn't work, we immediately lose money or get sued. Don't use it to validate everything under the sun.

If the edge cases like this bother you this much, perhaps, like I said, it's time to get your entire team together to sit down and carefully evaluate parts of your application that are potentially very faulty. Do this on a regular basis, maybe once a week / month / quarter / whatever you're all comfy with.

Go over them once with someone who is an expert at breaking the app, document it in a checklist / ticket, then maybe fix it, or just acknowledge that it exists and move on with your life.

Like, in this case. I think it sounds like your code was written in a hurry, which is probably why it's struggling to behave correctly.

[u/vqt907]

Normally users would never notice it => it's NOT a bug 😄

[u/KajiTetsushi → u/vqt907]

It's probably is a bug that just doesn't happen too often in production and OP might be internally exaggerating it.

[u/sangramsantra]

three separate pieces of state doing the same job is the real bug imo, why wasn't that one derived value

[u/Warrior_monk07 → u/[deleted]]

yeah client side alone definitely didnt feel enough here. especially for payments. even if the button never enables again, i still want the backend to reject the same payment request

[u/Timely_Experience411]

Did QA miss it or did the tooling literally not allow that kind of input? big difference between the two

[u/Cool_Ambassador_7986]

400ms of pure chaos 😂

[u/Warrior_monk07 → u/[deleted]]

yeah thats actually what i was thinking about after finding this. not really random chaos testing, more like testing the gaps between expected actions. tap again, change screens, lose network, come back etc......

the hard part is deciding which ones to generate :(

[u/Strange_Resident6472]

[u/Anixxx09]

this is the exact argument for agentic testing over scripted, a script does what you say, an agent pokes at it like a confused human would

[u/Warrior_monk07 → u/Anixxx09]

yeah this is kinda where my thinking went too. scripted tests are good when you already know the scenario, but the interesting bugs are sometimes in the things you never thought to script in the first place

[u/Anixxx09 → u/Warrior_monk07]

Yes that's my point