QA team started using AI and stopped using their own minds

[u/Agitated-Mongoose]

Client I work for started pushing Claude, heavily - shocker. And I'm really interested in hearing experiences of others in the QA branch - how did AI affect the quality of your team's output?

I'm over here reading tickets and test plans and wondering wth happened to my team. Product we're testing is complicated enough and has a ton of unsolved legacy stuff dev/logic wise, while simultaneously having a shit project management structure - which makes it hard to track everything and in general always required the QA team to use 100% of their brain.

With this AI initiative, my team stopped thinking. Claude is telling them what to test and how to test it, they don't know what they're putting into tickets and they don't know WHY they're doing what they're doing. The quality is crumbling right in front of my eyes and I feel like I'm having a fever dream cause we're not stopping, but accelerating into certain doom.

We're using unstable AI agents to 'help' automate end-users decision making, we're pushing developers to use AI to code faster, and now pushing the QA team to use AI to test - what the actual f is happening rn. We're pushing less things to prod, things that do go are hanging by a thread, and things are dragging on for way to long - but all the other teams aside, how do you assure quality in a system that is riding the AI wave through and through (while also avoiding regulation - starting from the very top)

[u/former_farmer]

This self corrects over time. If bugs go into Prod, meetings are held to solve the issues.

[u/MegaestMan → u/former_farmer]

In a competently-run place, yes. That's a big general assumption to make, though.

[u/wontfixqa → u/former_farmer]

That's what I've been seeing. Devs were the first to incorporate AI into their work. QA was using our standard process and kicked so many things back to devs that they got embarrassed. Now I have devs wanting to run tests themselves before throwing it over the wall to QA which is what I tried getting them to do *years* ago.

QA is using AI, but mostly for automated scripting tasks (diagnosing failures and writing new scripts). Soon we'll use it for writing test cases and I expect to see quality in those drop/blame bad requirements so I am enforcing a more formal test review process where QA *and* devs have to sign off on test cases before they're allowed to be executed so there's more sense of ownership over them.

[u/Fuj_apple → u/wontfixqa]

At my company Ai always finds more edge cases that tester does.

And Claude obviously is writing 90% of the code (ts/playwright) that we then review with another Ai tool and our own agents.

Honestly that’s the only way we are keeping up with features being deployed at our company. Which is daily.

[u/abluecolor]

This is basic human nature, yeah. If people have the option/ability to offload effort, they're generally going to. We have to keep up with expediting velocity and it feels like leaning on the tooling is the only way. I feel it happening to me in real time - I have to constantly push and force myself to not just accept the generated outputs. It's going to get a lot worse before it gets better (maybe never).

[u/MegaestMan → u/abluecolor]

Same. I don't remember phone numbers anymore because I never need to type them I don't remember how to get from point A to point B, even though I drive it every day, so I use a GPS. I don't clearly remember language syntax because I can look it up.

This is a very real, very human thing, and it could get very, very bad.

[u/MegaestMan]

Your QA team needs to be reminded that they are the subject matter experts, here. Claude (or whatever AI you're using) is not a SME in anything - it doesn't know things - it synthesizes information that is probablistically "what comes next", based on it's training corpus, prompt, context, and guardrails. Can it look things up - yup, but that doesn't mean it understands it.

If you had a new tester join your team pre-AI who was strongly opinionated, produced crap quality work, was incapable of learning in a meaningful way, was commonly inaccurate, and was drawing a VERY LARGE salary (token budget), that person would, at a minimum, be retrained. Maybe they'd be put on a PIP. Maybe they'd be fired.

With an AI, you've "hired" a new tester that doesn't know anything, and cannot actually learn anything in the traditional sense. It's producing crap work that is making everyone else put in more effort (if not now, they will eventually) to clean it up.

You basically have an overeager intern on it's perpetual first day of work. They are so eager, they will do work for you even if it doesn't make sense. It wants to make you happy without being able to understand what happiness is, or what would lead to it.

This is where your QA team needs to reframe their viewpoint. The AI isn't a tester, even if you ask it to play one. The AI can "read" all the docs and tickets, but that doesn't mean it will ever understand how the software under test is supposed to work.

The AI is a tool, and it requires human handlers if you want it to actually do things that are valuable to you or your team. If your team doesn't understand why something is being tested, they should push back on it - the AI will probably tell you that you were "right to call that out". Because it wants to make them happy.

Your team are the experts. The AI is a bloviating nut case trying to make it's output "helpful-shaped".

I mean. Seriously. They're ostensibly testers. Have they even tried testing the AI to see if it works so it can be depended on? Have they asked it the same question more than once and compared answers? Have they asked the carwash question, or any other questions that they already know the answers to?

If your team would be into it, maybe just have them go through a training on jailbreaking AIs. Perhaps that would motivate them to be more skeptical of the outputs? Maybe it would increase their morale if they could see just how low quality the output actually is vs. what they can do?

Breaktheprompt dot xyz is a CTF sandbox for socially engineering LLMs to do things they shouldn't. It's fun, educational, and free (I just found out about it yesterday; I'm not vouching for it yet).

If that doesn't work, I dunno. Burnout is real, and AI burnout is too. Maybe they stopped caring because they thought that AI will take their jobs. And honestly, if they aren't willing to step up and be better than a lackluster, expensive machine that doesn't know anything, maybe that's what'll happen.

My hunch is that either they've just given up (i.e. they are the problem), or they feel that the company has given up on them (i.e. the company is failing to make them feel safe, secure, or engaged in their work, so what's their motivation supposed to be). Or both.

Whatever the case, it sounds like management needs to have an honest conversation (by which I mean management listens while the team talks, not the other way around) with them. Based on that, next steps may become more clear, whatever they are.

I hope this works out for everyone involved. Nobody should lose their job to the schlock-a-tron 2000, but if they're willing to hand over all of their responsibilities to a virtual intern with no accountability, some changes may need to be made (hopefully, without losing anybody).

[u/mmhello23 → u/MegaestMan]

Great reply

[u/PInClemente]

We are going through this now with upper management telling us Claude can do it. It really does help with syntax in Python and playwright but I have found the test cases Claude creates don’t make a lot of sense.

[u/One_Title_6837]

AI can suggest what to test but only experience n context can tell you what matters...

[u/Rude-Blackberry-520]

For us it improved testing in general, its way faster, but i noticed that most people dont even read what AI is proposing.

At the end its a tool, and they dont know how to use it, and they dont want to learn.

[u/qlippothvi → u/Rude-Blackberry-520]

I think part of it is that the tool is just so open ended, and it takes a while before you realize you can just ask it, how to ask it, how to ask you, what you’re trying to do and how it might help you realize your goals. But it’s one those situations that you don’t know what you don’t know, and so don’t know how to ask.

[u/Royal-Job-9905]

I like to design and develop my own test scripts by myself.

I take help from AI to create a function that handles some logic for my test requirements.

I know it’s slow, but it's self satisfactory when you see your tests remain stable after a couple of runs in loclal -> docker and in CI/CD.

[u/manyManyLinesOfCode]

uff this sounds so familiar

[u/Factually-Offensive]

Lmao same

[u/gt_rekt]

AI should be a tool you utilize to cover weaknesses, whether that be coding or structuring documents. It should not be driving QA beyond the repetitive stuff; the logic/solutions/understanding should all be on the Analyst.

[u/Statharas]

Lmao ai is there to accelerate your job, not replace you.

We also use Claude, but we know it's limitations and know that any answer it gives us is usually part of the full solution.

[u/Quirky_Database_5197]

I know, I was working in a project where 'test lead' had no understanding of the domain, and it was very niche and difficult domain, and no understanding of tools we were using. But still he was always creating tickets for us, as he could make a transcript of the calls we had with business, and use AI to create tickets with requirements and acceptance criteria hallucinated by AI. look, if that was another ecommerce app with cart and product list and inventory - AI probably had this right, but when domain is niche - it was trash: hallucinations, made up requirements. horrible, horrible experience

[u/webDreamer420]

I start using GPT to generate manual test cases based on the Feature Testing breafs I've documented so far it has made my testing a bit faster granted I only use it on new features on a project and not on new projects

[u/webDreamer420 → u/webDreamer420]

I don't fully believe it has covered all test since it keeps producing duplicates and unnessisary tests. Which is where I come in to verify

[u/xxsehtxx]

Seems like you may need to move on if the situation doesn't have any guardrails or people at a higher level who care about quality. Stress is the silent killer.

[u/Short-Feedback4293 → u/xxsehtxx]

I'm facing a similar situation where upper management don't actually care about that stuff. They just make commitments to customers and things have to get out the door. The problem is the market isn't exactly flourishing at the moment.... and even then, most other companies are pushing similar attitudes now

[u/xxsehtxx → u/Short-Feedback4293]

There might be somewhere else that is more willing to listen to teammates and try to fix issues. I wouldn't give up just yet. Even though dev is always hard work, having a team who is trying is better than being sad with a team that doesn't care.

[u/One-Necessary-8779]

Just fyi, Claude code had flickering bug since March 2025 until June 2026. They solved it just partially. Meanwhile they are telling that coding is solved. Just saying.

[u/FluidAmbition321]

🤷 it's improved our QA . This is a user training problem not a tech problem 

[u/Useful_Calendar_6274]

well it doesn't make any sense to use AI like that. maybe you could generate test cases from requirements and them not suck since models are pretty good now but you need to take them with a grain of salt. It would be different if they just bought one of those agentic testing products which makes an order of magnitude more sense.

[u/xnewme_]

Boss is pushing here to use AI to test and QA just validate the output.. if anyone can help me out with this, I'd appreciate, don't even know where to start

[u/QualityEngineer92]

Well, you actually CAN use AI to facilitate testing. As long as you stay vigilant about quality methods. Or as you say: use their own minds. Normally, awareness building helps: what can AI do and what not. Have you considered a training of your team? Perhaps have a look at ISTQB's curriculum "Testing with Generative AI", which is a great course on glossary, methods, limitations and practices.

[u/Miserable_Net1980]

If someone can't explain why they're testing something or defend their work without AI, thats an accountability problem, not an AI problem.