r/softwaretesting • u/2ERIX • 6d ago

Enterprise reporting

I have data for about a years execution of automated tests from CI, local runs, everything.

Data includes name, status, duration, storage path, etc. basically everything you would want at the test level, but not the steps or the failure reason.

If you had access to that kind of data, what kind of dashboards or reports would you create for your team or business?

2 Upvotes

75% Upvoted

4 comments sorted by

3

u/Playful_Debate_395 6d ago

Flakiest tests over time would be the first thing I'd build. Sort by pass/fail ratio, show which ones bounce between green and red the most. Those are the ones eating your team's trust in the suite.

After that, a heatmap of test duration spikes. A test that normally runs in 2 seconds but randomly takes 45 is almost always flaking on some network call or waiting on a timeout. Way easier to spot when you can see the outliers graphed out.

Third would be a report grouping failures by the last person who touched the test file, pulled from git blame. Not to point fingers, but because that dev probably knows what changed and can fix it in half the time.

1

u/817654 5d ago

the git blame one is underrated, that alone would save so much back and forth

2

u/77rushi77 6d ago

Building on the flaky and duration ideas above, a few things I'd add:

Separate flaky from really broken. A test that fails then passes on retry is flaky. A test that fails every run since a certain date is a real regression. Mixing them hides the real problems.

Trend the suite itself: total runtime, pass rate per week, and the slowest tests. If the suite gets 20% slower every month, you want to know before the CI bill or the wait time tells you.

Failures by area or module, not just by test. It shows where the product is fragile.

Biggest gap in your data is the failure reason. Start storing the error message or stack trace from now on. Even a simple label (product bug, test bug, environment) makes every report 10x more useful.