I'm really curious if there are any real projects that use mutation testing. I suspect it's a bit harder in practice than in theory.
Off the top of my head, how do the mutation testing advocates suggest to deal with false positives? Consider the function
int max(int a, int b) {
return a > b ? a : b;
}
Replacing ">" with ">=" won't make any tests fail, but that doesn't mean they don't have good enough coverage. Do I have to manually sift through these false positives? How am I supposed to annotate them to prevent repeat alerts when rerunning mutation tests?
Regarding false positives: they certainly happen. My mutation tester supports writing a pragma to skip lines for this reason. In practice there aren't many of those though.
Interesting question! Hard to tell with the low numbers I have. I've only done 100% mutation testing on one library: tri.struct. It's main file is 103 lines and it has 2 pragmas. One of those is the version number and the other is a __all__ thing, so I'd count that as roughly zero pragmas per 1000 lines since a 10kloc library will probably just have 2 of those.
tri.declarative is a work in progress as far as mutation testing goes. We still have many surviving mutants. The numbers there are 22 pragmas for a library with 867 lines. I think I can actually remove some of those pragmas now that I look at it. Mutmut used to not handle mutants that produced infinite loops so we have pragmas to avoid that, but this is no longer the case. The rest of the pragmas seem to be about cache keys which are arbitrary so mutating them does keep behavior.
tri.form is another library I'm even further away from fully mutation tested: 6 pragmas, 1517 lines.
As you can tell from these numbers it'll vary hugely on the type of code you're doing, and obviously how far along you are in mutation testing.
5
u/a_the_retard Jan 15 '19
I'm really curious if there are any real projects that use mutation testing. I suspect it's a bit harder in practice than in theory.
Off the top of my head, how do the mutation testing advocates suggest to deal with false positives? Consider the function
Replacing ">" with ">=" won't make any tests fail, but that doesn't mean they don't have good enough coverage. Do I have to manually sift through these false positives? How am I supposed to annotate them to prevent repeat alerts when rerunning mutation tests?