{"id":914,"date":"2010-06-17T00:00:00","date_gmt":"2010-06-17T00:00:00","guid":{"rendered":"https:\/\/test.simple-talk.com\/uncategorized\/mutation-testing\/"},"modified":"2021-05-11T15:56:32","modified_gmt":"2021-05-11T15:56:32","slug":"mutation-testing","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/development\/dotnet-development\/mutation-testing\/","title":{"rendered":"Mutation Testing"},"content":{"rendered":"

What is Mutation Testing?<\/h2>\n

How do you know that you can trust your unit tests?\u00a0 How do you know that they’re really telling you the truth?\u00a0 If they don’t find a bug, does that really mean that there aren’t any?\u00a0 What if you could test your tests?<\/p>\n

Mutation testing \u00a0is one way that you can test those tests.\u00a0 Mutation testing involves deliberately altering a program’s code, then re-running a suite of valid unit tests against the mutated program.\u00a0 A good unit test will detect the change in the program and fail accordingly.<\/p>\n

There has recently been a resurgence of interest in Mutation testing, which was actually first conceived of by Richard Lipton in 1971, resulting in mutation testers appearing on many different platforms.\u00a0 Some notable examples are Java’s \u03bcJava<\/a>, Ruby’s Heckle<\/a>, C++’s Insure++<\/a>, and of course, C#’s Nester<\/a>.<\/p>\n

How Does Mutation Testing Work?<\/h2>\n

In a nutshell, mutation testing works in the following way.\u00a0 First you start with a piece of production code that’s well covered by unit tests.\u00a0 Once you’ve verified that all tests pass for a given piece of code it’s time to apply your mutation to your target assembly.\u00a0<\/p>\n

The extent of the mutation that you apply to your code can span many levels; some of the more common mutations simply involve replacing a logical operator with its inverse.\u00a0 For example, ==<\/code><\/b> can become !=<\/code><\/b> while <<\/code><\/b> can become >=<\/code><\/b>.\u00a0 In more complex mutations you may go so far as to rework the order of execution of code or even remove some lines of code completely.\u00a0 However, as mutations of this level can often cause compiler errors, it’s often easier to initially stick with the simpler mutations that I’ve mentioned.<\/p>\n

After your code has been mutated, it’s time to re-run your original suite of unit tests against it.\u00a0 If your tests are well written, any test that covers the mutated program code should fail.\u00a0 However, if your tests succeed in spite of the mutated program code then your tests are creating false positives and need to be revisited.<\/p>\n

Building Your Own Mutation Tester<\/h2>\n

Although there are a few mutation testers available for .NET, the best way to learn any technique is by writing your own.\u00a0 Over the next few paragraphs, we’ll walk through building our own simple mutation tester to test the test suite of a simple program.<\/p>\n

Something to Mutate<\/h2>\n

But before we can build our mutation tester, we need a program to test.\u00a0FirstBank, available here<\/a>, is a simple banking library we can use as our guinea pig.<\/p>\n

\"1061-MT1.JPG\"<\/p>\n

Let’s take a moment to familiarize ourselves with FirstBank<\/b>.\u00a0 First, we’ll note that there are two projects in the solution.\u00a0 The first project, FirstBank.Model<\/code><\/b> contains the production code for a simple Account<\/code><\/b>, AccountManager<\/code><\/b>, and an exception to represent when an account has insufficient funds to perform an action.\u00a0 There’s also an interface defined for an interest rate service.\u00a0 The second project, FirstBank.Model.Tests<\/code><\/b>, simply contains the unit tests which target the first project.<\/p>\n

Let’s take a closer look at the Account<\/code><\/b> class.<\/p>\n

The Account<\/code><\/b> class has two methods-Withdraw()<\/code><\/b> and Deposit()<\/code>,<\/b> as well as a single property-Balance<\/code><\/b>.\u00a0 The Deposit()<\/code><\/b> method simply accepts an amount from the caller and adds it to the account’s current balance.\u00a0 The Withdraw()<\/code><\/b> method accepts an amount from the caller and deducts it from the account’s current balance.\u00a0 However, the Withdraw()<\/code><\/b> method also has a check to ensure that the caller isn’t attempting to withdraw more than the account’s current balance.\u00a0 In the event that the caller asks for more than the current balance an exception of type InsufficientFundsException<\/code><\/b> is thrown.<\/p>\n

public void Withdraw(decimal amount)\r\n{\r\n\u00a0\u00a0\u00a0 if (amount > Balance)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 throw new InsufficientFundsException(amount, Balance);\r\n\u00a0\r\n\u00a0\u00a0\u00a0\u00a0Balance -= amount;\r\n}\r\n<\/pre>\n
Now that we’re familiar with the key parts of the FirstBank solution, let’s move onto our mutation tester.<\/p>\n
MutantPower<\/h2>\n
MutantPower is a very<\/i> simple mutation tester that simply inverts any Boolean conditionals that it finds in the code.\u00a0 I’ve intentionally kept MutantPower simple for two reasons.\u00a0 First, MutantPower is only really intended to be run against the FirstBank solution therefore it only needs to handle the few branching operations it encounters within that solution.\u00a0 Secondly, the simpler we keep the MutantPower codebase the easier it’ll be for us to talk about.\u00a0 MutantPower relies heavily on the Mono Project’s<\/a> Cecil<\/a> library to handle both traversing the object graph and the IL manipulation of .NET assemblies.\u00a0 You can download the source code for MutantPower here<\/a>.<\/p>\n
\n
Why Cecil?<\/p>\n
 Cecil was created as part of the Mono Project, which is an open source implementation of the .NET framework. Cecil is used heavily in many of Mono’s own development tools such as the Mono debugger and Gendarme, which is an FxCop-like static analysis tool. Although Cecil is actually an open source implementation of .NET’s Reflection.Emit library and has very similar capabilities, I tend to prefer Cecil over Reflection.Emit since it offers some more advanced functionality. In addition, since it’s not necessary for Cecil to actually load the target assembly into memory, Cecil tends to perform noticeably faster than Reflection.Emit.<\/p>\n<\/div>\n
MutantPower begins by loading the target assembly, passed as a command line parameter, into a ModuleDefinition<\/code><\/b> object.\u00a0 It then iterates through all of the types it finds in the module.\u00a0 In our example, these are types such as Account<\/code><\/b>, AccountManager<\/code><\/b>, and InsufficientFundsException<\/code><\/b>.\u00a0 Note that we ignore any type which is actually an interface, such as IInterestRateService<\/code><\/b>.\u00a0 The reason for this is that although interfaces do have members, those members do not have bodies so they have no IL to manipulate.\u00a0 Therefore interfaces are of little interest to a mutation tester.<\/p>\n
\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 var module = ModuleDefinition.ReadModule(inputAssembly);\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 foreach (var typeDefinition in module.Types)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if (typeDefinition.IsInterface)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 continue;\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 foreach (var methodDefinition in typeDefinition.Methods)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 foreach (var instruction in methodDefinition.Body.Instructions)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 . . .\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0}\r\n<\/pre>\n
 <\/p>\n
\"1061-MT2.JPG\"<\/p>\n
Within each type we also iterate though that type’s methods.\u00a0 This allows us to examine all methods of a class, regardless of visibility.\u00a0 What about properties?\u00a0 In C# properties are actually just syntactic sugar around specially named methods.\u00a0 So, in the IL world in which we’re working now, properties simply show up as just another method.\u00a0 For example, when we reflect over the Account<\/code><\/b> class’s Balance<\/code><\/b> property we see that the property actually exists in IL as two separate methods named get<\/code><\/b>_Balance()<\/code> and set<\/code><\/b>_Balance(Decimal)<\/code>.\u00a0<\/p>\n
Finally, within each method we iterate though the IL statements that comprise its body.\u00a0 You may have noticed that the FirstBank solution contained no abstract classes.\u00a0 However, if we were dealing with a solution that did we would want to take special care here to ignore any abstract methods.\u00a0 Just as interfaces contain no method bodies and thus are of little interest to a mutation tester, abstract methods are also missing method bodies and would be of just as little interest.<\/p>\n
Note the code…<\/p>\n
if (instruction.OpCode == OpCodes.Brtrue_S)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 instruction.OpCode = OpCodes.Brfalse_S;\r\n\u00a0\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0else if (instruction.OpCode == OpCodes.Brfalse_S)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0instruction.OpCode = OpCodes.Brtrue_S;\r\n<\/pre>\n
This is the heart of our mutator.\u00a0 This section of code replaces any instance of if (true)<\/code><\/b> with if (false)<\/code><\/b> and vice versa.\u00a0 Obviously this is a very simple mutator, but if we wanted to extend this to handle some of the other mutations mentioned in the beginning of the article, such as inverting logical comparisons, this is where we could do it.<\/p>\n
Finally, after all of the Boolean operators found have been inverted we save the newly mutated assembly back out to disk.\u00a0<\/p>\n
Running Your New Mutation Tester<\/h2>\n
Before we begin mutating our assembly, we must first make sure our code is in a good state to begin with.\u00a0 To do this, we need to run our unit tests against our original, un-mutated assembly.<\/p>\n
\"1061-MT3.JPG\"<\/p>\n
As you can see, all of our tests pass without incident.\u00a0 Now that we know we’re starting from a clean slate, it’s time to mutate.<\/p>\n
Our mutator accepts the assembly to be mutated as a command line argument which means that we can call the mutator like so.<\/p>\n
\"1061-MT4.JPG\"<\/p>\n
Once the mutator has completed you should have two assemblies in your output folder, FirstBank.Model.dll<\/i> and FirstBank.Model_backup.dll<\/i>.\u00a0 FirstBank.Model_backup.dll<\/i> is actually your original unharmed assembly whereas FirstBank.Model.dll<\/i> is your assembly after the mutations have been applied.\u00a0 It’s important that the mutated version of the assembly retain the same name as the original assembly.\u00a0 This is so that unit tests that were compiled against it can find the mutated assembly to retest.<\/p>\n
If we examine both assemblies using Red Gate’s .NET Reflector\u00c2\u00ae<\/a>, it becomes easy to spot the differences between the mutated version and the original version of the assembly.\u00a0 Looking in the Account.Withdraw()<\/code><\/b> method of the original assembly (FirstBank.Model_backup.dll<\/i>) we can see the code is still checking for insufficient funds and throwing the InsufficientFundsException<\/code><\/b> as expected.\u00a0 However, if we examine the same method in our newly mutated assembly (FirstBank.Model.dll) we see that the code is actually doing the opposite of what we would expect.\u00a0 In fact, the InsufficientFundsException<\/code><\/b> is only thrown if the amount that the account holder has requested is less than or equal to their current balance!<\/p>\n
\"1061-MT5.JPG\"<\/p>\n
You may be wondering why inverting the Boolean operators true<\/code><\/b> and false<\/code><\/b> resulted in the operator ><\/code> <\/b>becoming <=<\/code><\/b>.\u00a0 To answer this question we’ll have to examine the underlying IL that results from the C# code above.<\/p>\n
\"1061-MT6.JPG\"<\/p>\n
Take a look at line L_0012 in both sections of IL.\u00a0 This is really the only line of code between the two versions of the Account.Withdraw()<\/code><\/b> method that exhibited any significant change.\u00a0 That change, as expected, was an inversion of the brtrue<\/code><\/b>.s<\/code> command to the brfalse.s<\/code><\/b> command (the .s<\/code> suffix simply denotes the value we’re evaluating is stored as an 8-bit short).<\/p>\n
A full explanation of IL is well beyond the scope of this article, but in a nutshell the brtrue.s<\/code><\/b> and brfalse.s<\/code><\/b> commands on line L_0012 simply evaluate the result of the op_GreaterThan()<\/code><\/b> command executed on line L_0008, which is stored in location 0.\u00a0 Then, based on whether this value is true or false, line L_0012 instructs the runtime to continue to line L_001b, which throws the InsufficientFundsException<\/code><\/b>, or to skip the exception completely and go directly to line L_0021, which continues with the execution of the balance deduction.<\/p>\n
The only difference between the two methods is that the original Withdraw<\/code><\/b>()<\/code> method skips throwing the exception if the results of the op_GreaterThan<\/code><\/b>()<\/code> operation are true (the balance is greater than the amount requested) whereas the mutated Withdraw<\/code><\/b>()<\/code> method skips throwing the exception if the results are false (the balance is less than or equal to the amount requested).<\/p>\n
The interesting part is that our mutation didn’t actually change the logical operator we used, but instead it changed the control flow resulting from the execution of that operator.<\/p>\n
Let’s see how this new mutation affects our previously 100% green test suite.<\/p>\n
After running our test suite against our mutated assembly we see that we now have a mixture of both passing and failing tests.\u00a0 Unlike traditional unit testing, we actually strive for 100% of our tests failing when doing mutation testing.\u00a0 Why?\u00a0 Because if our tests failed after the code was mutated then we know they’re serving us well as regression tests by detecting changes in logic.<\/p>\n
\"1061-MT7.JPG\"<\/p>\n
Let’s start by looking at which tests failed.<\/p>\n
AccountTest.Can_withdraw_money_from_an_account<\/code><\/p>\n
This test failed because the Withdraw() method threw an InsufficientFundsException even though the caller was trying to withdraw less than their current account balance.\u00a0 However, this is exactly the behavior our mutated code now prescribes so this test successfully caught the mutation.<\/p>\n
AccountTest.Will_throw_InsufficientFundsException_when_trying_to_ withdraw_more_than_account_balance<\/code><\/p>\n
This test also failed but for the opposite reason than the test above.\u00a0 In this test the caller was able to withdraw more than their account balance without triggering an exception.\u00a0 Although this likely made the caller quite happy, their bank was likely less than thrilled with this new ‘feature’.\u00a0 This test also appears to be behaving exactly how we would expect it to..<\/p>\n
AccountManagerTest.Can_transfer_money_from_one_account_to_another<\/code><\/p>\n
Finally, a test in the AccountManagerTest<\/code><\/b> suite failed as well.\u00a0 This may seem a bit unexpected at first since we only mutated code in the Account.Withdraw()<\/code><\/b> method.\u00a0 However, once we examine the AccountManager.TransferFunds()<\/code><\/b> method the reason becomes obvious.\u00a0 The TransferFunds()<\/code><\/b> method actually calls the Account.Withdraw()<\/code><\/b> method.\u00a0 This means that this test failed for the same reason as the AccountTest.Can_withdraw_money_from_an_account<\/code> method above.<\/p>\n
In the vernacular of mutation testing, we would say that the mutant we introduced in the Account.Withdraw() method was ‘killed’ by these three tests.<\/p>\n
Now let’s take a look at the tests that passed.<\/p>\n