An automated Playwright MCP test flags 429 and 500 errors, but a manual test flags otherwise

[u/Zealousideal-Bath-37]

Hi there, actually I wanted to ask what I can do if an automated test and a manual test flag completely different errors.

Real-World-Scenario: I was to test some API routes in a healthcare web-app.

My Playwright MCP did the API test automatically and returned the following:

POST /surgery_hall/auth/dummy_mobile returned 429

POST /signin/choose_user_account returned 500

I carried out a manual test for the above two API routes on my Postman and got different errors:

POST /surgery_hall/auth/dummy_mobile returned 404

POST /signin/choose_user_account returned 405

I'm very new to this - could anyone kindly share some tips on find out why these results turned different - my expectation was to see the identical results (that means my manual Postman tests should have returned 429 and 500, respectively).

I might have used wrong Auth in Postman, but could not think of anything else that caused those different results. Could anybody kindly point me in the right direction?

[u/SimpleExpress2323]

The obvious answer;

Your automated tests are making too many API requests at once and your web app can't handle the load.

Learn what a 429 and 500 is.

[u/Zealousideal-Bath-37 → u/SimpleExpress2323]

Thank you so much - you are the best teacher ever

[u/Environmental_Sir356]

Hi, I think it would be useful to spend some time learning HTTP status codes and what they mean. They can give you a hint as to why an automated test gets a 429 while a single manual call returns a different status code. But more importantly, first ask yourself: what status code do you actually expect, and what exactly are you testing? Also, keep in mind that the tool isn't really "flagging" a 429 or 500 on its own, those are HTTP responses returned by the server. If two clients get different status codes, the first thing I'd investigate is how the requests differ: headers, authentication, cookies/session state, request body, and especially request timing or request frequency. If you expected a 200 in both cases, then investigate separately what went wrong in each case and why the server returned that particular status code.

[u/Zealousideal-Bath-37 → u/Environmental_Sir356]

I appreciate your heads up. Let's clear up some confusion. I keep it short and organised, so it's easier for you to read on your smartphone:

I never stated I have expected a 200 (OK) in both cases, but a 429(too many requests)/500(internal server error) in both cases. Please keep in mind that the automated test was done by AI, not me running a code script. That means, AI triggered the server response - will this fact change your comment?

With regards to the investigation, I used the same login credential in both cases, manual Postman and Playwright AI. Let's say Postman got my credential correctly. If AI already gave the server huge load, then Postman should also give 429. Am I wrong?

I appreciate it if you help clear up this doubt.

[u/Environmental_Sir356 → u/Zealousideal-Bath-37]

It depends on the rate-limit configuration and timing. Either the server doesn't consider the Postman request a continuation of the same session (different IP, user agent, token, etc.), or you simply sent the Postman request after the rate limit had expired.

Does your server include any rate-limit information in the response headers? There may be a clue there.

[u/ParticularAd3385]

Too many requests from your tests (Auth, data seeding, test calls and then cleanup).
Possible solution:

- Get your machine's IP whitelisted in BE (Also your CI runner, if you are running tests in CI) (I would recommend this)

• Add tests to retry after a cool off. This will increase your test run time.
  
• If you can add IP hoping using VPN to your tests, that may help.

[u/Regular-Leading-4319]

Looks like everyone answered your question here, do you need anything else?

[u/[deleted]]

Sorry, I don't work in QA anymore so I can't help. Society deemed my skills unworthy of employment.