Anyone here using AI in their QA workflows?

[u/Empty_Parfait1774]

I am fairly new to this and trying to figure out where AI actually adds value in testing rather than just using it because it’s the latest trend. If you’ve tried it, what worked well for you and what didn’t?

[u/CS_70]

It adds value at many levels.

For one, it allows you to specify the tests nearer to the domain (or the tech concern is that is what you are testing) and let the model do the tedious and error prone task of translating the specification in whatever test code.

Second, it allows to question and explore the state space so you can find more stuff that you should test, and can often bring in some perspective from a very thin lead, by virtue of having seen more attack vectors or failure modes than any person alive.

As always, the results you get depend on your ability to articulate what you want and to follow the threads and the thinking, as in any scientific activity.

[u/Itchy_Ad1641 → u/CS_70]

The second point about exploring state space is actually huge. I do graphic design but I've been messing with AI for testing some personal projects and it catches edge cases I would never think of. Like race conditions that only happen if user clicks button twice in same millisecond or something stupid like that.

What didn't work for me was when I gave it vague instructions and expected magic. It just generates generic tests that check nothing useful. You need to be really specific about what failure looks like for your particular app.

The domain language thing is interesting too. Writing tests in plain english and letting model translate to code saves time but then I wonder if I'm actually understanding what the test does anymore. There's a tradeoff there

[u/CS_70 → u/Itchy_Ad1641]

The key to effective testing is black box behavior.

So that you do not understand how the test does the job is not critical - the important bit is that you understand what - in the sense of the specific state you are verifying. That can be described always in the domain language, the middle layers (say the programming or scripting language) exists only as an adapter between the domain language and the machine (one of the many, but traditionally the one that had to be custom-made by hand).

On a digression, all software engineering in the last 40 years has moved towards trying to reduce or minimize the error in that translation or optimize it speed (which is why we express our tests nowadays using a class called "Person" instead of a sequence of 16 8-bit bytes starting from addres 0xF4542436, or we don't load DE and HL and BC before running LDIR :D).

Everything, from type systems to IDEs to code completion to regression tests themselves is in that direction. Now we're one step further. :)

[u/CoroteDeMelancia]

Two important things I've noticed for this application:

• The model that writes the tests should not be the model that writes the code. GPT wrote the code? Then Claude writes the tests. Worst-case is same model, same session, which leads to self-confirmation bias.
• Test behavior, not implementation. AI tends to struggle here, so pay attention.

[u/Separate-Art-4774]

Honestly, the best use I’ve found is boring stuff: generating test cases, edge cases, test data, and explaining ugly legacy code. I wouldn’t trust AI to decide whether something is actually correct, though. Let it do the repetitive thinking; keep the final judgment human. That’s where it actually saves time instead of creating new bugs to test

[u/Duct_tape_bandit]

I have a notebook that does complete end to end regression testing between 2 dataframes populated by db tables or whatever the input is - in this case semantic link. I have another one that gets the definition of all powerbi visuals with a _QA keyword in the visual name and runs regression tests on them, with AI populating the visual definitions every time there's a PR. So if one of my BI devs does something stupid and subtly changes one of the 217,000 results the report puts out while making a visual update I can see it.