Guidelines: Test Plan

Topics

• Overview
• Identifying Requirements for Tests
  • Requirements for Functional Tests
  • Requirements for Performance Tests
  • Requirements for Reliability Tests
• Assess Risk and Establish Test Priorities
  • Assess Risk
  • Determine Operational Profile
  • Establish Test Priorities
• Test Strategy
  • Type of Test and Objective
  • Test Stage
  • Technique
  • Completion Criteria
  • Special Considerations

Overview

The purpose of the test plan is to communicate the intent of the testing activities. It is critical that this document be created as early as possible. Generating this artifact early in one of the first iterations of the Elaboration phase would not be too early. It may be desirable to develop the test plan iteratively, adding sections as the information is available.

Care should be taken in clearly communicating the scope of testing, the requirements for test, the test strategies, and the resource needs. This information identifies the purpose and boundaries of the test effort, what will be tested, how it will be tested, and what resources are needed for testing. Stating this information clearly, will expedite the review, feedback, and approval of the test effort.

At the outset of the project, a test plan identifying the overall intended testing for the project should be created, called the "Master Test Plan." As each iteration is planned, a more precise "Iteration Test Plan" is created (or several test plans, organized by type of test), containing only the data (requirements for test, test strategies, resources, etc.) that pertain to the iteration. Alternately, this information may be included in the Iteration Plan, if it does not make the iteration plan too difficult to manage or use. 

Below are some guidelines to better identify and communicate the requirements for test, test risks and priorities, and test strategies.

Identifying Requirements for Test

Requirements for test identify what will be tested. They are the specific target of a test. There are a few general rules to apply when deriving requirements for test:

• The requirement for test must be an observable, measurable behavior. If the requirement for test cannot be observed or measured, it can't can be assessed to determine if the requirement has been satisfied.
• There is not a one-to-one relationship between each use case or supplemental requirement of a system and a requirement for test. Use cases will often have more than one requirement for test, while some supplemental requirements will derive one or more requirements for test and others will derive none (such as marketing or packaging requirements).

The requirements for test may be derived from many sources, including use cases, use-case models, supplemental specifications, design requirements, business cases, interviews with end-users, and the software architecture document. All of these should be reviewed to gather information that is used to identify the requirements for test.

Requirements for Functional Tests

Functional requirements for test, as their name implies, are derived from descriptions of the target-of-test's functional behaviors. At a minimum, each use case should derive at least one requirement for test. A more detailed list of requirements for test would include at least one requirement for test for each use case flow of events.

Requirements for Performance Tests

Performance requirements for test are derived from the target-of-test's specified performance behaviors. Typically, performance is stated as a measure of response time and/or resource usage, as measured under various conditions, including

• different workloads and/or system conditions
• different use cases
• different configurations

Requirements for performance are described in the Supplementary Specifications. Review these materials, paying special attention to statements that include the following:

• statements of time, such as response time or timing profiles
• statements indicating that a number of events or use cases must occur in a stated period of time
• statements comparing the behavior of one item to another
• statements comparing the application behavior on one configuration to that of another
• operational reliability (mean time to failure or MTTF) over a period of time
• configurations or constraints

You should derive at least one requirement for test for each statement in the specification which reflects information such that listed above.

Requirements for Reliability Tests

Reliability requirements for test are derived several sources, typically described in Supplementary Specifications, User-Interface Guidelines, Design Guidelines, and Programming Guidelines.

Review these artifacts and pay especial attention to statements that include the following:

• statements of reliability or resistance to failure, run-time errors (such as memory leaks)
• statements indicating code integrity and structure (compliance to language and syntax)
• statements regarding resource usage

At least one requirement for test should be derived from each statement in the artifacts that reflects information listed above.

Successful testing requires that the test effort successfully balance factors such as resource constraints and risks. To accomplish this, the test effort should be prioritized so that the most important, significant, or riskiest use cases or components are tested first. To prioritize the test effort, a risk assessment and operational profile are performed and used as the basis for establishing the test priority.

The following sections describe how to determine test priority.

Assess Risk and Establish Test Priorities

Identifying the requirements for test is only part of identifying what will be tested. Prioritizing what will be tested and in what order should also be performed. This step is done for several reasons, including:

• to ensure the test efforts are focused on the most appropriate requirements for test
• to ensure the most critical, significant, or riskiest requirements for test are addressed as early as possible
• to ensure that any dependencies (sequence, data, etc.) are accounted for in the testing

There are three steps to assessing risk and establishing the test priorities:

• Assess Risk
• Determine Operational Profile
• Establish Test Priority

Guidelines for each of these three steps are provided below:

Assess Risk

Establishing the priority for test begins with the assessment of risk. Use cases or components that pose the greatest risk due to failure or have a high probability of failure should be among the first use cases tested.

Begin by identifying and describing the risk magnitude indicators that will be used, such as:

• H - high risk, not tolerable. Severe external exposure. The company will suffer great financial losses, liability, or un-recoverable loss of reputation.
• M - medium risk, tolerable, but not desirable. Minimal external exposure, the company may suffer financially, but there is limited liability or loss of reputation.
• L - low risk, tolerable. Little or no external exposure, company has little or no financial loss or liability. Company's reputation unaffected.

After identifying the risk magnitude indicators, list each use case or component in the target-of-test. For each use case or component in your list, identify a risk magnitude indicator, and justify (in a brief statement) the value you selected.

There are three perspectives that can be used for assessing risk:

• Effect - the impact or consequence of a specified use case (requirement, etc.) failing
• Cause - an undesirable outcome caused by the failure of a use case
• Likelihood - the probability of a use case failing.

Select one perspective, identify a risk magnitude indicator and justify your selection. It is not necessary to identify an indicator for each risk perspective. However, it is suggested that, if a low indicator was identified, try evaluating the item from a different risk perspective to ensure the item is really a low risk.

Below are greater details on assessing risk by these three perspectives.

Effect

To assess risk by Effect, identify a condition, event, or action and try to determine its impact. Ask the question:

"What would happen if ___________?"

For example:

• "What would happen if while installing the new software, the system runs out of disk space?"
• "What would happen if the Internet connection is lost during an inquiry transaction?"
• "What would happen if the Internet connection was lost during a purchase transaction?"
• "What would happen if the user enters an unexpected value?"
  

Below is a sample justification matrix for these items:

Cause

Assessing risk by Cause is the opposite of by Effect. Begin by stating an undesirable event or condition, and identify the set of events that could have permitted the condition to exist. Ask a question such as:

"How could ___________ happen?

For example:

• "How could only some of the files be on the system and not all the registry entries made?"
• "How could a transaction not be reflected properly in the central database?
• "How could the billing cycle statement reflect only some of records in the database that fulfill the desired criteria?"

Below is a sample justification matrix for these items:

Likelihood

Assessing risk by Likelihood is to determine the probability that a use case (or component implementing a use case) will fail. The probability is usually based on an external factors such as:

• Failure rate(s) and / or density
• Rate of change
• Complexity
• Origination / Originator

It should be noted, that when using this risk perspective, the risk magnitude indicators are related to the probability of a failure, not the effect or impact the failure has on the organization as was used in assessing risk by Effect and Cause.

Correlations between these factors and the probability of a failure exist, as identified below:

For example:

• Installing the new software
• "Historically we've found many defects in the components used to implement use cases 1, 10, and 12, and our customers requested many changes in use case 14 and 19."

Below is a sample justification matrix for these items:

Determine Operational Profile

The next step in assessing risk and establishing a test priority is to determine the target-of-test's operational profile.

Begin by identifying and describing the operational profile magnitude indicators that will be used, such as:

• H - used quite frequently, many times per period or by many actors or use cases.
• M - used frequently, several times per period or by several actors or use cases.
• L - infrequently used or used by very few actors or use cases.

The operational profile indicator you select should be based upon the frequency a use case or component is executed, including:

• the number of times ONE actor (or use case) executes the use case (or component) in a given period of time, or
• the number of ACTORS (or use cases) that execute the use case (or component)

Typically, the greater the number of times a use case or component is used, the higher the operational profile indicator.

After identifying the operational profile magnitude indicators to be used, list each use case or component in the target-of-test. Determine an operational profile indicator for each item in your list and a state your justification for the indicator value. Information from the workload analysis document (See Artifact: Workload Analysis Document) may be used for this assessment.

Examples:

• Installing new software
• Ordering items from the on-line catalog
• Customers inquiring about their order on-line after order is placed
• Item selection dialog

Establish Test Priority

The last step in the assessing risk and establishing a test priority is to establish the test priority.

Begin by identifying and describing the test priority magnitude indicators that will be used, such as:

• H - must be tested
• M - should be tested, will test only after all H items are tested
• L - might be tested, but not until all H and M items have been tested

After identifying the test priority magnitude indicators to be used, list each use case or component in the target-of-test. Determine a test priority indicator for each item in your list and a state your justification. Below are some guidelines for determining a test priority indicator.

Consider the following when determining the test priority indicators for each item:

• the risk magnitude indicator value you identified earlier
• the operational profile magnitude value you identified earlier
• the actor descriptions (are the actors experienced?, tolerant of work-arounds?, etc.)
• contractual obligations (will the target-of-test be acceptable if a use case or component is not delivered?)

Strategies for establishing a test priority include:

• Use the highest assessed factor (risk, operational profile, etc.) value for each item as the overall priority.
• Identify one assessed factor (risk, operational profile, other) as being the most significant and use that factor's value as the priority.
• Use a combination of assessed factors to identify the priority.
• Using a weighting schema where individual factors are weighed, and their values and priority calculated based upon the weight.

Examples:

• Installing new software
• Ordering items from the on-line catalog
• Customers inquiring about their order on-line after order is placed
• Item Selection Dialog

Priority when the highest assessed value is used to determine priority:

Priority when the highest assessed value for one factor (Risk) is used to determine priority:

Priority when a weighting value is used to calculate the priority:

(Note: in the matrix below, H = 5, M = 3, and L = 1. A Total Weighted value greater than 30 is a High priority test item, values between 20 and 30 inclusive are a Medium priority, and values less than 20 are Low).

Test Strategy

The Test Strategy describes the general approach and objectives of a specific test effort.

A good test strategy should contain the following:

• type of test to be implemented and its objective
• stage in which test will be implemented
• technique
• measurement and criteria used to assess test results and test completion
• any special considerations that affect the test effort described in the test strategy

Type of Test and Objective

State clearly the type of test being implemented and the objective of the test. Explicitly stating this information reduces confusion and minimizes misunderstandings (especially since some tests may look very similar). The objective should state clearly why the test is being executed.

Examples:

"Functional Test. The functional test focuses on executing the following use cases implemented in the target-of-test, from the user interface."

"Performance Test. The performance test for the system will focus on measuring response time for use cases 2, 4, and 8 - 10. For these tests, a workload of one actor, executing these use cases without any other workload on the test system will be used."

"Configuration Test. Configuration testing will be implemented to identify and evaluate the behavior of the target-of-test on three different configurations, comparing the performance characteristics to our benchmark configuration."

Test Stage

Clearly state the stage in which the test will be executed. Identified below are the stages in which common test are executed:

Technique

The technique should describe how testing will be implemented and executed. Include what will be tested, the major actions to be taken during test execution, and the method(s) used to evaluate the results.

Example:

Functional Test:

• For each use case flow of events, a representative set of transactions will identified, each representing the actions taken by the actor when the use case is executed.
• A minimum of two test cases will be developed for each transaction; one test case to reflect the positive condition and one to reflect the negative (unacceptable) condition.
• In the first iteration, use cases 1 - 4, and 12 will be tested, in the following manner:
  • Use Case 1:
    • Use Case 1 begins with the actor already logged into the application and at the main window, and terminates when the user has specified SAVE.
    • Each test case will be implemented and executed using Rational Robot.
    • Verification and assessment of execution for each test case will be done using the following methods:
      • Test script execution (did each test script execute successfully and as desired?)
      • Window Existence, or Object Data verification methods (implemented in the test scripts) will be used to verify that key windows display and specified data is captured / displayed by the target-of-test during test execution.
      • The target-of-test's database (using Microsoft Access) will be examined before the test and again after the test to verify that the changes executed during the test are accurately reflected in the data.

Performance Test:

• For each use case, a representative set of transactions, as identified in the workload analysis document will be implemented and executed using Rational Suite PerformanceStudio (vu scripts) and Rational Robot (GUI scripts).
• At least three workloads will be reflected in the test scripts and test execution schedules including the following:
  • Stressed workload: 750 users (15 % managers, 50 % sales, 35 % marketing)
  • Peak workload: 350 users (10 % managers, 60 % sales, 30 % marketing)
  • Nominal workload: 150 users (2 % managers, 75% sales, 23 % marketing)
• Test scripts used to execute each transaction will include the appropriate timers to capture response times, such as total transaction time (as defined in the workload analysis document), and key transaction activity or process times.
• The test scripts will execute the workloads for one hour (unless noted differently by the workload analysis document).
• Verification and assessment of execution for each test execution (of a workload) will include:
  • Test execution will be monitored using state histograms (to verify that the test and workloads are executing as expected and desired)
  • Test script execution (did each test script execute successfully and as desired?)
  • Capture and evaluation of the identified response times using the following reports:
    • Performance Percentile
    • Response Time

     

Completion Criteria

Completion criteria are stated to for two purposes:

• identify acceptable product quality
• identify when the test effort has been successfully implemented

A clear statement of completion criteria should include the following items:

• function, behavior, or condition being measured
• method of measurement
• criteria or degree of conformance to measurement

Example 1

• All planned test cases have been executed
• All identified defects have been addressed to an agreed upon resolution
• All planned test cases have been re-executed and all known defects have been addressed as agreed upon, and no new defects have been discovered

Example 2

• All high priority test cases have been executed.
• All identified defects have been addressed to an agreed upon resolution.
• All Severity 1 or 2 defects have been resolved (status = fixed or postponed).
• All high priority test cases have been re-executed and all known defects have addressed as agreed upon, and no new defects have been discovered.

Example 3

• All planned test cases have been executed.
• All identified defects have been addressed to an agreed upon resolution.
• All Severity 1 or 2 defects have been resolved (status = verified or postponed).
• All high priority test cases have been re-executed and all known defects have addressed as agreed upon, and no new defects have been discovered.

Special Considerations

This section should identify any influences or dependencies which may impact or influence the test effort describe in the test strategy. Influences might include:

• human resources (such as availability or need for non-test resources to support / participate in test)
• constraints, (such as equipment limitations or availability, or the need / lack of special equipment)
• special requirements, such as test scheduling or access to systems

Examples:

• Test databases will require the support of a database designer / administrator to create, update, and refresh test data.
• System performance testing will use the servers on the existing network (which supports non-test traffic). Testing will need to be scheduled after hours to ensure no non-test traffic on the network.
• The target-of-test must synchronize the legacy system (or synchronization simulated) for full functional testing to be implemented and executed

Rational Unified Process