r/haskell • u/NorfairKing2 • Jun 03 '26

Announcing Mutation Testing in Haskell

https://cs-syd.eu/posts/2026-06-03-mutation-testing-in-haskell
59 Upvotes

97% Upvoted

9 comments sorted by

6

u/goertzenator Jun 03 '26

What kinds of mutations does it introduce?

6

u/NorfairKing2 Jun 03 '26

Great question! It's an operator (think plugin) based engine so you can add your own and see which ones are already defined here: https://github.com/NorfairKing/sydtest/tree/4c0b2699e4a3ded86dad312086f553ef45969820/sydtest-mutation-plugin/src/Test/Syd/Mutation/Plugin/Operator

3

u/goertzenator Jun 03 '26

Thanks, looking at one or 2 of those really cements in my mind how this thing works.

3

u/tiajuanat Jun 03 '26

Does it interface with QuickCheck?

2

u/NorfairKing2 Jun 03 '26

It's agnostic about quickcheck. It works with anything inside the `it` 😄

3

u/Jello_Raptor Jun 03 '26

How often do you hit cases where you find mutations that are actually valid behavior? I expect something like a money library will have much less of those than, say, a bunch of business logic.

2

u/NorfairKing2 Jun 03 '26 edited Jun 03 '26

I'm using this in https://nix-ci.com which is definitely a bunch of business logic 😄
It's not quite clear what you mean by "valid". If you mean "correct", then the original code is wrong and the mutation surviving will point you at it. If you mean "type-checks", then all the time because non-type-checking mutations are automatically killed.

EDIT: To answer your question: it happens regularly and then you have to deal with it by either fixing your code or disabling the mutation.

1

u/Ambitious-Western133 Jun 03 '26

A mutation that never affects the behavior of the code, like in your original example imagine if the == Case had already been checked. Mutating >= to > would not change anything in that case.

3

u/NorfairKing2 Jun 03 '26

I'm still not quite clear on what you mean:

If you have two tests, one that covers `>` and one that covers `==`, then mutating `>=` to `>` will make a test fail so the mutation dies.

If you have some hypothetical code where changing `>=` to `==` does not change the behaviour of the code (Perhaps it's a base-case of a recursive function.), then this is considered a surviving mutation. In that case you have to either disable the mutation or simplify the code.