Guidelines: Maintaining Automated Test Suites

Topics

• Introduction
• Abstraction helps manage complexity
• Another example
• Focusing test improvement
• Throwing away tests

Introduction

Like physical objects, tests can break. It's not that they wear down, it's that something's changed in their environment. Perhaps they've been ported to a new operating system. Or-more likely-the code they exercise has changed in a way that correctly causes the test to fail. Suppose you're working on version 2.0 of an e-banking application. In version 1.0, this method was used to log in:

public boolean login (String username);

In version 2.0, the marketing department has realized that password protection might be a good idea. So the method is changed to this:

public boolean login (String username, String password);

Any test that uses login will fail. It won't even compile. Since not much useful work can be done without logging in, not many useful tests can be written that don't use login. You might be faced with hundreds or thousands of failing tests.

These tests can be fixed by using a global search-and-replace tool that finds every instance of login(something)and replaces it with login(something, "dummy password"). Then arrange for all the testing accounts to use that password, and you're on your way.

Then, when marketing decides that passwords should not be allowed to contain spaces, you get to do it all over again.

This kind of thing is a wasteful burden, especially when-as is often the case-the test changes aren't so easily made. There is a better way.

Suppose that the tests originally did not call the product's login method. Rather, they called a library method that does whatever it takes to get the test logged in and ready to proceed. Initially, that method might look like this:

public boolean testLogin (String username) {
  return product.login(username);
}

When the version 2.0 change happens, the utility library is changed to match:

public Boolean testLogin (String username) {
  return  product.login(username, "dummy password");
}

Instead of a changing a thousand tests, you change one method.

Ideally , all the needed library methods would be available at the beginning of the testing effort. In practice, they can't all be anticipated-you might not realize you need a testLogin utility method until the first time the product login changes. So test utility methods are often "factored out" of existing tests as needed. It is very important that you perform this ongoing test repair, even under schedule pressure. If you do not, you will waste much time dealing with an ugly and un-maintainable test suite. You might well find yourself throwing it away, or being unable to write the needed numbers of new tests because all your available testing time is spent maintaining old ones.

Note: the tests of the product's login method will still call it directly. If its behavior changes, some or all of those tests will need to be updated. (If none of the login tests fail when its behavior changes, they're probably not very good at detecting defects.)

Abstraction helps manage complexity

The previous example showed how tests can abstract away from the concrete application. Most likely you can do considerably more abstraction. You might find that a number of tests begin with a common sequence of method calls: they log in, set up some state, and navigate to the part of the application you're testing. Only then does each test do something different. All this setup could-and should-be abstracted into a single method with an evocative name such as readyAccountForWireTransfer. By doing that, you're saving considerable time when new tests of a particular type are written, and you're also making the intent of each test much more understandable.

Understandable tests are important. A common problem with old test suites is that no one knows what the tests are doing or why. When they break, the tendency is to fix them in the simplest possible way. That often results in tests that are weaker at finding defects. They no longer test what they were originally intended to test.

Another example

Suppose you're testing a compiler. Some of the first classes written define the compiler's internal parse tree and the transformations made upon it. You have a number of tests that construct parse trees and test the transformations. One such test might look like this:

/* 
 * Given
 *   while (i<0) { f(a+i); i++;}
 * "a+i" cannot be hoisted from the loop because 
 * it contains a variable changed in the loop.
 */
loopTest = new LessOp(new Token("i"), new Token("0"));
aPlusI = new PlusOp(new Token("a"), new Token("i"));
statement1 = new Statement(new Funcall(new Token("f"), aPlusI));
statement2 = new Statement(new PostIncr(new Token("i"));
loop = new While(loopTest, new Block(statement1, statement2));
expect(false, loop.canHoist(aPlusI))

This is a difficult test to read. Suppose that time passes. Something changes that requires you to update the tests. At this point, you have more product infrastructure to draw upon. In particular, you might have a parsing routine that turns strings into parse trees. It would be better at this point to completely rewrite the tests to use it:

loop=Parser.parse("while (i<0) { f(a+i); i++; }");
// Get a pointer to the "a+i" part of the loop. 
aPlusI = loop.body.statements[0].args[0];
expect(false, loop.canHoist(aPlusI));

Such tests will be much easier to understand, which will save time immediately and in the future. In fact, their maintenance costs are so much lower that it might make sense to defer most of them until the parser is available.

There's a slight downside to this approach: such tests might discover a defect in either the transformation code (as intended) or in the parser (by accident). So problem isolation and debugging may be somewhat more difficult. On the other hand, finding a problem that the parser tests miss isn't such a bad thing.

There is also a chance that a defect in the parser might mask a defect in the transformation code. The chance of this is rather small, and the cost from it is almost certainly less than the cost of maintaining the more complicated tests.

Focusing test improvement

A large test suite will contain some blocks of tests that don't change. They correspond to stable areas in the application. Other blocks of tests will change often. They correspond to areas in the application where behavior is changing often. These latter blocks of test will tend to make heavier use of utility libraries. Each test will test specific behaviors in the changeable area. The utility libraries are designed to allow such a test to check its targeted behaviors while remaining relatively immune to changes in untested behaviors.

For example, the "loop hoisting" test shown above is now immune to the details of how parse trees are built. It is still sensitive to the structure of a while loop's parse tree (because of the sequences of accesses required to fetch the sub-tree for a+i). If that structure proves changeable, the test can be made more abstract by creating a fetchSubtree utility method:

loop=Parser.parse("while (i<0) { f(a+i); i++; }");
aPlusI = fetchSubtree(loop, "a+i");

expect(false, loop.canHoist(aPlusI));

The test is now sensitive only to two things: the definition of the language (for example, that integers can be incremented with ++), and the rules governing loop hoisting (the behavior whose correctness it's checking).

Throwing away tests

Even with utility libraries, a test might periodically be broken by behavior changes that have nothing to do with what it checks. Fixing the test doesn't stand much of a chance of finding a defect due to the change; it's something you do to preserve the test's chance of finding some other defect someday. But the cost of such a series of fixes might exceed the value of the test's hypothetically finding a defect. It might be better to simply throw the test away and devote the effort to creating new tests with greater value.

Most people resist the notion of throwing away a test-at least until they're so overwhelmed by the maintenance burden that they throw all the tests away. It is better to make the decision carefully and continuously, test by test, asking:

1. How much work will it be to fix this test well, perhaps adding to the utility library?
2. How else might the time be used?
3. How likely is it that the test will find serious defects in the future? What's been the track record of it and related tests?
4. How long will it be before the test breaks again?

The answers to these questions will be rough estimates or even guesses. But asking them will yield better results than simply having a policy of fixing all tests.

Another reason to throw away tests is that they are now redundant. For example, early in development, there might be a multitude of simple tests of basic parse-tree construction methods (the LessOp constructor and the like). Later, during the writing of the parser, there will be a number of parser tests. Since the parser uses the construction methods, the parser tests will also indirectly test them. As code changes break the construction tests, it's reasonable to discard some of them as being redundant. Of course, any new or changed construction behavior will need new tests. They might be implemented directly (if they're hard to test thoroughly through the parser) or indirectly (if tests through the parser are adequate and more maintainable).

Rational Unified Process