What is Mutation Testing?
Mutation testing is a powerful tool for ensuring the quality of your tests. The goal of mutation testing is to ensure that your tests are comprehensive enough to detect any changes to the code.
It works by introducing small changes to your code, called mutations, and then running your tests to see if they detect the changes. If the tests fail, then the mutation has been detected and the tests are considered to be of high quality.
If the tests fail to detect a mutation, then it’s likely that the code is not being tested thoroughly enough. This can lead to bugs and other issues in the code that could have been avoided if the tests had been more exhaustive.
A Demonstration of a Mutation
To demonstrate the benefit of mutation testing, let’s look at a simple example. Consider the following code:
int Add(a, b)
{
return a + b;
}
If we run our tests without mutation testing, they should all pass since the code works as expected. Now let’s introduce a mutation to this code by changing the + operator to a - operator:
int Add(a, b)
{
return a - b;
}
Now when we run the unit tests, we should see failures since the behavior of the code has changed. This is a good indication that our tests are of high quality. If no tests fail, then the mutation testing step should also fail, and we should add more thorough unit tests to catch this mutation in future.
How can I Add Mutation Testing to my Workflow?
Scrolling through our code and manually altering lines of code and then running the tests clearly isn’t scalable, so to introduce mutation testing into our workflow, we need to use some tooling. Most of these tools tend to be language specific as they need to be able to understand the source code and alter it in some way.
The open-source tool Stryker supports a number of languages and is what I use for C#. The installation is very simple, as they support the dotnet tool CLI command, and it is also highly configurable and easy to add as a step to your CI pipeline.
To run locally against a C# project, use the following commands:
dotnet tool install -g dotnet-stryker
dotnet stryker
And that’s it! You will receive a report of any mutations that survived, allowing you to cover that code with more robust tests.
Do I need Mutation Testing if I Enforce Code Coverage?
Code coverage is a metric used to measure the amount of code that is tested. While code coverage can be a useful tool to identify areas of code that need more testing, it should not be used as a measure of quality or as a goal to strive for.
It is now common to see tools deployed to CI pipelines which force quality gates upon new code. For example, it is possible to block a PR being merged into the main branch until the new code has met a certain percentage of coverage from unit tests. My opinion is that enforced code coverage percentages typically lead to a false sense of security and can be detrimental to the quality of code.
Developers may be tempted to write tests that are designed to meet the coverage percentage, rather than tests that are designed to ensure the code is working correctly. This can lead to tests that are not comprehensive enough to catch all potential bugs, leaving the code vulnerable to errors. This is why I recommend mutation testing, it will weed out these types of tests and help increase confidence.
Developers may also be tempted to write code that is designed to meet the coverage percentage, rather than code that is designed to be efficient and maintainable. This can lead to code that is difficult to read and understand, as well as code that is more prone to errors.
Finally, enforcing a certain code coverage percentage can lead to a lack of focus on other important aspects of code quality, such as readability, maintainability, and performance. These are all important aspects of code quality that should be taken into consideration when writing code, and should not be overlooked in favour of meeting a certain code coverage percentage. Mutation testing will help find low-value tests, as well as areas of your code that require more coverage.
Summary
I hope from this post, you can see how valuable mutation testing can be, and how easy it is to add to your workflow, both locally and within your CI pipelines. Code coverage reports are becoming more common as part of the developer workflow and I hope to see mutation testing follow this trend, as my opinion is that it is a much more valuable measurement of how extensive a testing suite is. Spread the word!
