Tool Mentor: Executing Test Suites Using Rational Robot

Purpose

This tool mentor describes how to use Rational Robot to execute Test Suites (playing back one or more Test Scripts), and to analyze the results of the test execution.

This section provides links to additional information related to this tool mentor.

• Execute Test Suite
• Execute Developer Tests

Overview

This tool mentor is applicable when running Microsoft Windows 98/2000/NT 4.0.

To execute the test scripts recorded or programmed using Rational Robot, play back the test script.

1. Play the script back
2. View the results in the TestManager log
3. Analyze verification point results with the Comparators

1. Play the script back

When you play back a script, Rational Robot repeats your recorded actions and automates the execution of the test. With automation, you can test each new build of your application faster and more thoroughly than by testing it manually. This decreases testing time and increases both coverage and overall consistency.

Test scripts are played back at several times during the test life cycle:

• Test Implementation - Play back the scripts to verify that they work as intended, using the same build of the application-under-test that you used to record. This verifies the baseline of expected behavior for the application-under-test.
• Test Execution and Regression - Play back the scripts to compare the latest build of the application-under-test to the baseline established during test implementation. Ongoing play back of all test scripts during the development cycle reveals any differences that may have been introduced into the application since the last build. These differences can be evaluated to determine if they are actual defects or deliberate changes.

Before playback, Rational Robot automatically compiles a Test Script if it has changed since it last ran.

For information on preparing the test environment for Test Script playback, see Tool Mentor: Setting up the Test Environment in Rational Robot.

To play back a Test Script:

1. Click the Playback Script button on the toolbar. The Playback dialog box will appear.
2. Type a Test Script name or select it from the list.

To change the Playback options, click GUI Options.

When finished, click OK. For information on playback options, see Tool Mentor: Setting up the Test Environment in Rational Robot.

3. Click OK to continue.
4. If the Specify Log Information dialog box appears, do the following:

• Select a Build from the list. To create a new Build, click the Build button on the right.
• Select a Test Log folder from the list. To create a new Test Log folder, click the Log Folder button on the right.
• Accept the default Test Log name, which is the same as the Test Script name, or type a new name.
• Click OK.

6. If a prompt appears asking if you want to overwrite the log, click Yes.

Note: To stop playback of a Test Script, press the F11 key. Rational Robot recognizes the F11 key only when playing back Object-Oriented commands. The F11 key does not stop Test Script playback during low-level actions.

2. View the results in the TestManager Test Log

After playback completes, use the TestManager Test Log to view the playback results, including verification point failures, procedural failures, aborts, and any additional playback information.

To control the Test Log information and display of the Test Log, you can set options in the Log tab in the GUI Playback Options dialog box:

• To update your project with the playback results, select Output playback results to log.
• To display the log automatically after playback, select View log after playback.

If this is not selected, you can display the Test Log after playback by clicking Tools > Rational Test > Rational TestManager.

If the playback of the Test Scripts resulted in any errors, the specific cause of the error must be identified.

Test Scripts which failed to execute sucessfully are identified in the Test Log by the word Fail (shown in red) in the Results column, or Warning (shown in yellow). There are several classes of execution failure to be addressed: fatal errors, script errors, and verification point failures. Fatal errors often indicate that there is a problems with the Test Environment Configuration rather than the Test Script itself. Investigate the Test Environment Configuration and isolate the failure and resolve the corresponding fault. Resolving Test Script errors may require some Test Script maintenance. Verification point failures (during test implementation) typically require changes to the verification point parameters or indicate a difference between the desired state of the Target Test Items when the script was recorded versus when it was played back. See the next section for analyzing verification points.

Additional information regarding the error condition may be viewed in the Log Event Properties dialog box. Open this box by selecting a Test Log event and clicking View > Properties.

For each Test Script in which a failure (or warning) occurred, review the information regarding the failure and if necessary, edit the Test Script (see Section 3. Edit the test script if necessary above). The Test Script (and Rational Robot) may be opened from the Test Log by clicking View > Script.

After editing the Test Script, the environment should be re-set to its appropriate initial state (see Tool Mentor: Setting Up the Test Environment in Rational Robot) and the Test Suites re-executed (See step: Play the script back).

3. Analyze the verification point results with the Comparators

Use the Comparators to analyze verification point results. To open a Comparator from the TestManager Test Log:

1. In the Log Event column, select a verification point and click View > Verification Point. The appropriate Comparator will appear. The Comparators may also be opened from Rational Robot by double-clicking a verification point in the Asset pane. However, when you open a Comparator this way, you can view only the baseline file.
2. Differences between the baseline and the actual results will be highlighted.
3. Review the differences and determine the appropriate course of action, including:
   (Note: the available actions will differ based upon the type of verification point.)

• Copy property to baseline - copy the current individual actual property and over-write the baseline property.
• Copy all properties to baseline - copy all the actual property values and over-write all the baseline properties.
• Edit individual property (menu, value, and so on) by double-clicking the property (menu, value, and so forth) and entering a different value.
• Editing the properties list (to identify which properties should be captured by the verification point).
• Create or modify masks.

4. When finished making the appropriate changes in the Comparator, click File > Save Baseline to save the changes, and click the Exit button to close the Comparator.

After editing the Test Script or verification points, the environment should be reset to its appropriate initial state (see Tool Mentor: Setting Up the Test Environment in Rational Robot) and the tests re-executed (See step: Play the script back).

Rational Unified Process