Coverage Analysis

What Is It?

Mutation testing involves running a project's tests after making a mutation. As an optimisation, Fettle will only run tests that are relevant to the code it's mutating, instead of the entire test suite.

To work out which tests are relevant to which bits of the code, Fettle does some analysis of your project's code coverage when it starts (aka Test Impact Analysis). This takes a little while but usually makes the overall process of mutation testing much quicker.

Support

Coverage analysis works by hooking in to the NUnit 3 API.

This means that it can't work if you're using a custom test runner command. If you use one Fettle will automatically skip coverage analysis.

Skipping It

By default, coverage analysis will be performed if possible. However, if you wish, you can disable this step via a command-line flag:

--skipcoverageanalysis (-s)

Why Skip It?

There are two main reasons why you might want to skip coverage analysis:

1. Memory usage
   There is an issue where Fettle can use a large amount of memory when doing coverage analysis. Skipping the analysis means that memory usage will be minimal, and pretty-much linear throughout the mutation testing process.

2. Reliance on Coverage
   When coverage analysis has been carried out, Fettle won't bother mutating code that is not covered by tests. This may be fine if, in addition to mutation testing, you're also keeping an eye on coverage metrics to spot un-tested code. But you may prefer for Fettle to highlight all un-tested parts of your code, especially if coverage is low.

Note that mutation testing can take a lot longer when coverage analysis is skipped.