TDD Beyond Basics : Mutation Testing

Objective

  • How to test your test

Discussion


It is a good idea to test your tests. You can verify that they detect the errors you think they detect by inserting those errors into the production code. Make sure they report errors in a meaningful way. You should also verify that your tests speak clearly to a person trying to understand your code. The only way to do this is to have someone who isn’t familiar with your code read your tests and tell you what they learned.
Tip by Gerard Meszaros in 97 Things a Programmer Should Know

What is Mutation Testing?


Mutation testing is the process re-writing code to flush out ambiguities in the code. These ambiguities can cause failures. Since these faults are often very subtle, it can easily pass testing and debugging and end up in production.

What Can be Mutated?


  • Changes if to unless, while to until etc.
  • Replace logical operator with its inverse. Ex: && with || etc.
  • Replace arithmetic operations with others. Ex: + with -, * with / etc.
  • Replace boolean relations with others. Ex: == becomes !=, < becomes > etc.
  • Replace true with false and vice-versa

Conditonals Boundary Mutator

alt text

Negate Conditionals Mutator

alt text

Remove Conditionals Mutator

Replace conditional statement with true.

if (a == b) {
 // do something 
}

becomes:

if true {
  // do something
}

Math Mutator

alt text

Increments Mutator

Replace += with -=.

Invert Negatives Mutator

The return values will be inverted. So a positive number will be changed to negative and vice-versa.

Mutation in Ruby Projects


Ruby has heckle and mutant gems for mutation testing.

gem install heckle --pre

Fails with :

ERROR:  Error installing heckle:
    ZenTest requires RubyGems version ~> 1.8. Try 'gem update --system' to update RubyGems itself.

Heckle is outdated and has installation problems because it depends on ruby gems 1.8. My Ruby gems version is 2.2. The alternative to Heckle is Mutant. I had difficulties running the Mutant gem with my project. Reading the tests for Heckle and Mutant gems does not provide any information on what gets mutated. The tests and the code are difficult to understand. These two gems has failed the Gerard Meszaros tip mentioned in the beginning of this article. So, let's do the mutation testing manually and see if our test fails to catch the bugs. Here is the code from previous article:

class OctalConverter
  def initialize(number)
    @number = number
  end

  def result
    octal = []
    until finished?
      digit = extract_octal_digit
      reduce

      octal.unshift(digit)
    end

    octal
  end

  private

  def extract_octal_digit
    @number % 8
  end

  def reduce
    @number = @number / 8 
  end

  def finished?
    @number == 0
  end
end

The mutations that I made manually are: changing 'until' to 'while', / to *, == to !=. The tests either failed or was caught in an infinite loop. So our tests break when the code is mutated.

References


  1. Mutation Testing
  2. Mutation Testing: A New Approach to Automatic Error-Detection
  3. Mutators


Related Articles


Create your own user feedback survey