Guidelines: Test Case

Topics

• Explanation
• Deriving Test Cases from Use Cases
• Deriving Test Cases from Supplementary Specifications
  • Deriving Test Cases for Performance Tests
  • Deriving Test Cases for Security / Access Tests
  • Deriving Test Cases for Configuration Tests
  • Deriving Test Cases for Installation Tests
  • Deriving Test Cases for other Non-Functional Tests
• Deriving Test Cases for Product Acceptance Tests
• Build Verification Test Cases for Regression Tests
• Defining Test Data for Test Cases

Explanation

Nothing has a greater influence on the project teams ability to ensure the stakeholder's satisfaction with the software, than the availability of a clear specification of what the stakeholders expectations are. With or without a good enough set of requirements specifications, test cases are one artifact that helps to reflect the expectations of the stakeholders, enabling those expectations to be verified and validated.

When a useful set of requirements are available, the test team needs to plan tests that will appropriately validate those requirements. Note that validating the software against the requirements may be done differently depending on the type of requirement. For example, executing the software to validate its functional and performance requirements may be done by a tester using automated test techniques, while verifying a configuration requirement such as the shutdown of the host computer system may need to be done using manual testing techniques.

Since you may not be able (or be responsible) to verify all requirements, it is important to focus on the most appropriate or critical requirements for the scope of the current work effort. The requirements you choose to verify will be a balance between the cost, risk, and necessity of having the requirement verified, and will generally be limited by the scope of the current iteration.

While requirements are an important source from which tests can be derived, they are not the only information source. In fact, in many cases they will be insufficient to provide a complete basis from which tests are developed. Test Cases should also be considered that are based on information sources such as risks, constraints, technologies, change requests (defects), faults and so on.
See Concept: Test Ideas for more information on how to come up with ideas from which tests can be derived.

Identifying test cases is useful for several reasons.

• Test cases can be used as the foundation on which to design and implement the actual tests. Time spent considering the test case helps to understand the design and implementation requirements better, and has the potential to save time in design and implementation activities.
• Some tests are particularly complex or detailed. Tests of this nature can benefit from careful consideration in advance before starting implementation of the test, and test case and test design artifacts are good tools for exploring those considerations.
• The "depth" of the testing is typically regarded as proportional to the number of tests. Greater confidence in the test process itself is often gained when the potential "depth" of testing can be reasoned about based on the number of test cases identified.
• One measure of the completeness of the test effort is based on monitoring coverage against some set of motivating elements. Coverage reporting can be based on measures such as the number test cases identified, and the number of tests implemented and / or executed against each test case, or the amount of effort expended against each test case.
• The scale and complexity of the test effort is to some extent proportional to the number of test cases. With a breakdown of test cases, the test effort can be reasoned about at a finer level of granularity.
• The kinds of test design and development, and the resources needed are in part governed by the number and complexity of the test cases.

However, there are some concerns worth considering regarding test cases:

• Not every test is complex enough to warrant the overhead of creating a test case artifact that needs to be reviewed and maintained: the test is simple enough that a short textual description is enough to convey what is required. In fact, the majority of test cases may fall in the category. Time spent documenting a vast number of simple test cases may result in time lost from more important testing activities.
• Some of the initial ideas you have for tests are subsequently proven to be flawed in some respect. This means that some of the test cases you initially identified based on those ideas will be abandoned. This reality means that any work you do documenting test cases in detail may be subsequently abandoned, and any reporting of coverage based on test cases needs to consider that situation. As such, it may be better to base test coverage reporting on the basis of higher-level concerns than test cases, and to treat test cases as internal test team artifacts used as required.

Test cases are often categorized or classified by the type of test or requirement for test they are associated with, and will vary accordingly. One heuristic for identifying test cases is to start by considering the following two perspectives:

• demonstrating the requirement has been achieved, often referred to as a positive test case,
• demonstrating the requirement is only achieved under the desired conditions, referred to as a negative test. This test case reflects unacceptable, abnormal, or unexpected conditions or data that the software may reasonably be subjected to.

Deriving Test Cases from Use Cases

Test cases for functional testing are derived from the target-of-test's use cases (see Artifact: Use Case). Test cases should be developed for each use-case scenario. The use-case scenarios are identified by describing the paths through the use case that traverse the basic flow and alternate flows start to finish through the use case.

In the diagram below, for example, each of the different paths through a use case reflecting the basic and alternate flows, are represented with the arrows. The basic flow, represented by the straight, black-line is the simplest path through the use case. Each alternate flow begins with the basic flow and then, dependent upon a specific condition, the alternate flow is executed. Alternate flows may rejoin the basic flow (alternate flows 1 and 3), may originate from another alternate flow (alternate flow 2), or may terminate the use case without rejoining a flow (alternate flows 2 and 4).

Sample Flows of Events for a use case

Following each possible path through the use case in the above diagram, the different use-case scenarios can be identified. Beginning with the basic flow and then combining the basic flow with alternate flows, the following use-case scenarios can be identified: 

Note: For simplicity, Scenarios 5, 6, and 8 only depict a single execution of the loop indicated by Alternate flow 3.    

Deriving the test cases for each scenario is done by identifying the specific condition that will cause that specific use-case scenario to be executed.

For example, suppose the use case depicted in the diagram above stated the following for Alternate Flow 3:

"This flow of events occurs if the dollar amount entered in Step 2 above, "Enter Withdraw Amount" is greater than the current account balance. The system displays a warning message and then rejoins the basic flow at Step 2 "Enter Withdraw Amount" above, where the bank customer can enter a new withdrawal amount."

With this information, you can begin to identify the test cases needed to execute the alternate flow 3:

Note: the test cases shown above are very simplistic since no other information was provided. Test cases are rarely this simple.

A more realistic example of deriving test cases from use cases is provided below:

Example:

Actors and use cases in an ATM machine.

The following table contains the basic flow and some alternate flows for Cash Withdrawal use case in the diagram above:

The following scenarios can be derived from this use case:

Note: For simplicity the loops in Alternate flows 3 and 6 (Scenarios 3 and 7), and combinations of loops have not been included in the table above.

For each of these seven scenarios, test cases need to be identified. Test cases can be identified and managed using matrices or decision tables. A common format is shown below, where each row represent an individual test case, and the columns identify test case information.  In this example, for each test case, there is a test case ID, Condition (or description), and all the data elements participating in the test case (as input or already in the database), and expected result.

To begin developing the matrix, start by identifying what data elements are required to execute the use-case scenarios. Then, for each scenario, identify at least test case that contains the appropriate condition to execute the scenario. For example, in the matrix below, V (valid) is used to indicate this condition must be VALID for the basic flow to execute and I (invalid) is used to indicate the condition which will invoke the desired alternate flow. In the table below, "n/a" indicates that this condition is not applicable to the test case.

In the matrix above, the six test cases execute the four scenarios. For the Basic Flow, test case CW1 above is known as a positive test case. It executes the Basic Flow path through the use case without any deviations. Comprehensive testing of the Basic Flow must include negative test cases to ensure that the Basic Flow is taken only when the conditions are correct. These negative test cases are represented by test cases CW2 - 6 (the shaded cell indicates the condition needed to execute the alternate flows). While CW2 - 6 are negative test cases for the Basic Flow, they are positive test cases for Alternate flows 2 - 4, and there is at least one negative test case each of these Alternate Flows (CW1 - the Basic Flow).

Scenario 4 is an example where having just one positive and one negative test case per scenario is not sufficient. To thoroughly test Scenario 4 - Incorrect PIN, at least three positive test cases (to invoke Scenario 4) are needed:

• the incorrect PIN is entered and there are tries left and this Alternate Flow rejoins the Basic Flow Step 3 - Enter PIN)
• the incorrect PIN is entered and there are no remaining tries left and this Alternate Flow then retains the card and terminates the use case.
• the CORRECT PIN is entered when there are no remaining tries left.  This Alternate Flow rejoins the Basic Flow at Step 5 - Enter Amount. 

Notice, that in the above matrix, no actual values were entered for the conditions (data). An advantage of creating the test case matrix in this manner is that it is easy to see what conditions are being tested. It is also very easy to determine if sufficient test cases have been identified, since you only need to look at Vs and Is (or as done here - shaded cells). Looking at the above table, there are several conditions for which there is no shaded cell, therefore, we are missing test cases, such as for Scenario 6 - No Account or Incorrect Account Type and Scenario 7 - Insufficient Account Balance.

Once sufficient test cases have been identified, they should be reviewed and validated to ensure accuracy, appropriateness, and eliminate duplicate, equivalent or otherwise redundant test cases. See Concepts: Test-Ideas List for more details. Also see the section Defining Test Data for Test Cases for additional details.

The test cases above are only a few of the test cases needed to verify the Cash Withdraw Use Case for this iteration. Other test cases needed include:

• Scenario 6 - No Account or Incorrect Account Type: Account not found or available
• Scenario 6 - No Account or Incorrect Account Type: Account does not allow withdraws
• Scenario 7 - Insufficient Account Balance: Amount requested greater than amount in account.

In future iterations, when other flows are implemented, test cases will be needed for:

• Invalid cards (card is reported lost, stolen, is not from an accepted bank, has a damaged stripe, etc.)
• Inability to read a card (card reader is jammed, off-line, or malfunctioning)
• Account is closed, frozen, or otherwise unavailable
• Amount in ATM is insufficient or incapable of creating requested amount (different than CW3, in that one denomination is out, but not all)
• Incapable of contacting banking system for approval
• Bank network goes off line, or power failure mid-transaction

When identifying functional test cases, ensure the following:

• sufficient test cases, positive and negative, have been identified for each use-case scenario 
• test cases address any business rules implemented by the use cases, ensuring that there are test cases inside, outside, and at the boundary condition / value for the business rule
• test cases address any sequencing of events or actions, such as those identified in the sequence diagrams in the design model, or user interface object states or conditions.
• test cases address any special requirements defined for the use case, such minimum/maximum performance, sometimes combined with minimum/maximum loads or data volumes during the execution of the use cases.

See the section Defining Test Data for Test Cases for additional guidance.

Deriving Test Cases from Supplementary Specifications

Not all requirements for a target-of-test will be reflected in functional requirements artifacts such as use-case specifications. Nonfunctional requirements, such as performance, security and access, and configuration requirements specify additional behaviors or characteristics of the target-of-test, and are often documented separately from the functional requirements. The Supplementary Specification is one of the primary sources for deriving test cases for these additional requirements.

Below are described guidelines for deriving these additional test cases:

• Deriving Test Cases for Performance Tests
• Deriving Test Cases for Security / Access Tests
• Deriving Test Cases for Configuration Tests
• Deriving Test Cases for Installation Tests
• Deriving Test Cases for other Nonfunctional Tests

Deriving Test Cases for Performance Tests

The primary input for performance test cases are the Supplementary Specifications which contain the nonfunctional requirements (see Artifact: Supplementary Specifications). Use the following guidelines when deriving test cases for performance test:

• ensure there is at least one test case identified for each statement in the supplementary specification which states a performance criteria. Performance criteria are usually expressed as time per transaction, number of transactions / users, or percentiles.
• ensure there is at least one test case identified for each critical use case. Critical use cases are those identified in the above statements and / or in the workload analysis document that must be evaluated using performance measures.
  (see Artifact: Workload Analysis Document)

As with test cases for functional tests, there will typically be more than one test case per usage scenario or requirement. It is common to define multiple test cases - for example, one that is below the performance threshold value (average transaction rate), another at the threshold value (high transaction rate), and a third test case above the threshold value (peak transaction rate).

In addition to the above performance criteria, ensure that you identify the specify conditions that affect response times, including:

• Size of the database - how many records exist?
• Workload - transaction patterns:
  • type, number and frequency of simultaneous end-user actions,
  • type, number, frequency and duration of simultaneous transactions being performed
• Environment characteristics (hardware, netware, software configuration)

A common practice is to capture test cases for performance tests in tabular matrices similar to those used for function test.

See the section Defining Test Data for Test Cases for additional details.

Here are some examples for the different types of Performance Tests:

For Load Test:

For Stress Test:

Deriving Test Cases for Security / Access Tests

Actors and use cases describe the interaction between external users of the system and the actions performed by the system to yield a value to a particular actor. Complex systems contain many actors and it is critical that we develop test cases to ensure that only those actors specified to execute the use cases can do so. This is especially true if there are differences in the use case flow of events based upon actor type.

For example, in the ATM use cases, different use case flow of events may be executed for the actor "Bank Customer" if their card and account is from the bank that owns the ATM versus the "Bank Customer" who uses a bank card (and account) from a competing bank, or tries to use a non-participating bank card.

Follow the same guidelines as listed above for functional test cases.

See the section Defining Test Data for Test Cases for additional guidance.

Example test cases for Security and Access:

Deriving Test Cases for Configuration Tests

In typical distributed systems there can be many allowed combinations of hardware and software that will be supported. Testing should be performed to verify that the target-of-test functions or performs acceptably in different configurations, such as with different operating systems, browsers, or CPU speeds. Furthermore, testing also needs to cover combinations of components to uncover defects that come from interactions of the different components, for example, ensuring that the version of DLLs installed by one application do not conflict the versions of the same DLLs expected by another application.

To derive test cases for configuration testing, use the following guidelines:

• Ensure there is at least one test case identifying each critical configuration. This is done by identifying the required hardware and software configurations for the target-of-test's environment and prioritizing the configurations, ensuring the most common ones are tested first, including:
  • Printer support
  • Network connections - local and wide area networks
  • Server configurations - server drivers, server hardware
  • Other software installed on the desktop and / or servers
  • Software versions for all installed software
• Ensure there is at least one test case for each configuration likely to have problems. These may include:
  • Hardware with the lowest performance.
  • Co-resident software that has a history of compatibility problems.
  • Clients accessing the server over slowest possible LAN/WAN connection.
  • Insufficient resources (slow CPU speed, minimum memory or resolution, disk space, etc.)

Deriving Test Cases for Installation Tests

Installation testing needs to verify that the target-of-test can be installed under all possible installation scenarios. Installation scenarios may include installing the target-of-test for the first time, or installing a newer version or build of the target-of-test (onto a machine containing the older version). Install testing should also ensure that the target-of-test performs acceptably when abnormal conditions are encountered, such as insufficient disk space.

The test cases should cover installation scenarios for the software including:

• Distribution media, for example, diskettes, CD-ROM, or file server.
• New installation.
• Complete installation.
• Custom installations.
• Upgrade installations.

Installation programs for client-server software have a specialized set of test cases. Unlike host-based systems, the installation program is typically divided between the server and the client. Therefore, it is important that installation testing perform the installation of all components of the target-of-test, including the client, middle tiers, and servers.

Deriving Test Cases for other Nonfunctional Tests

Ideally, you should find all the necessary input to derive test cases in the Use-Case Model, the Design Model, and the Supplementary Specification artifacts. It is, however, not uncommon that you at this point need to complement what is found there.

Examples would be:

• Test cases for Operational Tests (to verify that the software works when in use for a "long time" between failures).
• Test cases that investigate performance bottlenecks, volume capabilities of the system, or stress the target-of-test to failure.

In most cases, you can find these test cases by creating variants or aggregates of the test cases you've derived from those previously identified.

Deriving Test Cases for Product Acceptance Tests

Product acceptance testing is the final test action prior to deploying the software. The goal of acceptance testing is to verify that the software is ready and can be used by the end-users to perform those functions and tasks the software was built to do. Product acceptance testing often involves more than execution of the software for readiness, it also involves all product artifacts delivered to the customer(s), such as training, documentation, and packaging. 

Deriving test cases for the software artifact(s) is done in the manner described in the sections above. Depending upon the degree and of formality of the product acceptance test, the test cases will either be the same or similar to those identified above (formal), or a subset (informal). Independent of the depth of the test cases, agreement on the test cases and product acceptance criteria should be reached at before product testing is implemented and executed.

Evaluating the non-software artifact(s) varies greatly dependent upon the artifact being evaluated. Refer to each specific non-software artifact's Guidelines and Checklists for information regarding what / how to evaluate it.

Build Verification Test Cases for Regression Tests

Regression testing compares two builds or versions of the same target-of-test and identifies differences as potential defects. It thus assumes that a new version should behave like an earlier one and ensures that defects have not been introduced as a result of the changes.

Ideally, you would like all the test cases in one iteration to be used as test cases in the later iterations. The following guidelines should be used to identify, design, and implement test cases that maximize the value of regression testing and reuse, while minimizing maintenance:

• Ensure the test case identify only the critical data elements (those needed to create / support the condition being tested
• Ensure each test case describes or represents a unique set of inputs or sequence of events that results in a unique behavior by the target-of-test
• Eliminate redundant or equivalent test cases
• Group together test cases which have the same target-of-test initial state and state of the test data

Defining Test Data for Test Cases

Once the test cases have been discussed and their is general agreement/ approval for them, the actual data values can be identified in more detail (e.g. in the test case implementation matrix), and the test data artifacts created.

See Guidelines: Test Data for additional information regarding defining and maintaining test data.

Rational Unified Process