Guidelines: Test
Design
Topics
Explanation 
Nothing has a greater effect on the end-user's satisfaction with the software than a clear view of what the end-user expects so that those expectations can be verified and validated. Test cases reflect the requirements that are to be verified. Verifying these requirements, however, may be done differently and by different testers. For example, executing the software to verify its function and performance may be done by a tester using automated test techniques, the shut-down sequence of a computer system may be done by manual test and observation, while market share and sales, (also product requirements), will be done by measuring product and competitive sales.
Since you may not be able to (or be responsible to) verify all requirements, it is critical for the success of your project to select the most appropriate or critical ones requirements for test. The requirements you choose to verify will be a balance between the cost, risk, and necessity of having the requirement verified.
Identifying the test cases is important for several reasons.
- Test cases form the foundation on which to design and develop Test Scripts.
- The "depth" of the testing is proportional to the number of test cases. Greater confidence in the quality of the product and test process is gained when the number of test cases increases, since each test case reflects a different scenario, condition, or flow through the product.
- A principal measure of the completeness of test is requirements-based coverage, based on the of the number test cases identified, implemented, and / or executed. A statement such as "95 percent of our critical test cases have been executed and verified" is more significant than stating "We're 95 percent of the way through our tests."
- The scale of the test effort is proportional to the number of test cases. With a comprehensive breakdown of test cases, the timing of succeeding stages of the test cycle can be more accurately estimated.
- The kinds of test design and development, and the resources needed are largely governed by the test cases.
Test cases are often categorized or classified by the type of test or requirement for test they are associated with, and will vary accordingly. Best practice is to develop at least two test cases for each requirement for test:
- a test case to demonstrate the requirement has been achieved, often referred to as a positive test case,
- another test case, reflecting an unacceptable, abnormal, or unexpected condition or data, to demonstrate that the requirement is only achieved under the desired condition, referred to as a negative test cases.
Deriving Test Cases for Unit Test
Unit testing requires testing both the unit's internal structure and its behavioral characteristics. Testing the internal structure requires a knowledge of how the unit is implemented, and tests based upon this knowledge are known as white-box tests. Testing a unit's behavioral characteristics focuses on the external observable behaviors of the unit without knowledge or regard its implementation. Tests based upon this approach are referred to as black-box tests. Deriving test cases based upon both approaches are described below.
White-Box
Tests
Theoretically, you should test every possible path through the code. Achieving such a goal, in all but very simple units, is either impractical or almost impossible. At the very least you should exercise every decision-to-decision path (DD-path) at least once, resulting in executing all statements at least once. A decision is typically an if-statement, and a DD-path is a path between two decisions.
To get this level of test coverage, it is recommended that you choose test data so that every decision is evaluated in every possible way. Toward that end, the test cases should make sure that:
- Every Boolean expression is evaluated to true and false. For example the expression (a<3) OR (b>4) evaluates to four combinations of true/false
- Every infinite loop is exercised at least zero times, once, and more than once.
Use code-coverage tools to identify the code not exercised by your white box testing. Reliability testing should be done simultaneously with your white-box testing.
Example:
Assume that you perform a structure test on a function member in the class Set of Integers. The test - with the help of a binary search - checks whether the set contains a given integer.
The member function and its corresponding flowchart. Dotted arrows illustrate how you can use two test cases to execute all the statements at least once.
Theoretically, for an operation to be thoroughly tested, the test case should
traverse all the combinations of routes in the code. In member, there
are three alternative routes inside the while-loop. The test case can
traverse the loop either several times or not at all. If the test case does
not traverse the loop at all, you will find only one route through the code.
If it traverses the loop once, you will find three routes. If it traverses twice,
you will find six routes, and so forth. Thus, the total number of routes will
be 1+3+6+12+24+48+..., which in practice, is an unmanageable number of route combinations.
That is why you must choose a subset of all these routes. In this example, you
can use two test cases to execute all the statements. In one test case, you
might choose Set of Integers = {1,5,7,8,11} and t = 3 as test
data. In the other test case, you might choose Set of Integers = {1,5,7,8,11}
and t = 8.
See Guidelines: Unit Test for additional information
Black-Box
Tests
The purpose of a black-box test is to verify the unit's specified behavior without looking at how the unit implements that behavior. Black-box tests focus and rely upon the unit's input and output.
Equivalence partitioning is a technique for reducing the required number of tests. For every operation, you should identify the equivalence classes of the arguments and the object states. An equivalence class is a set of values for which an object is supposed to behave similarly. For example, a Set has three equivalence classes: empty, some element, and full.
Use code-coverage tools to identify the code not exercised by your white box testing. Reliability testing should be done simultaneously with your black-box testing.
The next two subsections describe how to identify test cases by selecting test data for specific arguments.
Test Cases based upon Input Arguments
An input argument is an argument used by an operation. You should create test cases by using input arguments for each operation, for each of the following input conditions:
- Normal values from each equivalence class.
- Values on the boundary of each equivalence class.
- Values outside the equivalence classes.
- Illegal values.
Remember to treat the object state as an input argument. If, for example, you test an operation add on an object Set, you must test add with values from all of Set's equivalence classes, that is, with a full Set, with some element in Set, and with an empty Set.
Test Cases based upon Output Arguments
An output argument is an argument that an operation changes. An argument can be both an input and an output argument. Select input so that you get output according to each of the following.
- Normal values from each equivalence class.
- Values on the boundary for each equivalence class.
- Values outside the equivalence classes.
- Illegal values.
Remember to treat the object state as an output argument. If for example, you test an operation remove on a List, you must choose input values so that List is full, has some element, and is empty after the operation is performed (test with values from all its equivalence classes).
If the object is state-controlled (reacts differently depending on the object's state), you should use a state matrix such as the one in the following figure.
A state matrix for testing. You can test all combinations of state and stimuli on the basis of this matrix.
See Guidelines: Unit Test for additional information
