Tool Mentor: Analyzing Test Results Using Rational Test RealTime
Purpose
This tool mentor describes how to analyze the results of test execution from
Rational Test RealTime. The value of testing early and often cannot be realized
if the results of the testing activity cannot be quickly and easily interpreted
and subsequently acted upon.
This section provides links to additional information related to this tool mentor.
Overview
Using both a source code insertion technology as well as source code analysis,
Rational Test RealTime is able to focus its component testing and runtime analysis
functionality directly at the source code level. A variety of features have
been added to ensure that fault isolation is quick and that it is easy and efficient
to work on the associated source code:
- source code can be opened and modified in the Test RealTime Text Editor.
- version control programs (e.g. Rational ClearCase) can be manipulated to
check-in/check-out versioned code.
- change management programs (e.g. ClearQuest) can be directly accessed for
defect submissions and enhancement requests
These features are used in conjunction with the analysis reports produced by
the test tools. Each key function of Test RealTime-component testing, system
testing, memory profiling, performance profiling, code coverage analysis, runtime
tracing, static metric generation-has a dedicated report. These reports
contain information extracted from a number of log files generated by source
code parsers, source code instrumentors, test script generators, test script
compilers, target deployment ports and report generators. These files can be
opened automatically by Test RealTime at the conclusion of a run, or can be
opened manually.
Each report is designed to be immediately interpretable to ensure problem diagnosis
and resolution is quick and efficient.
Types of Reports
Report analysis for each Test RealTime feature will be discussed. Where relevant,
report differences due to programming language used will be discussed:
To open these reports using the Project Browser tab in the Project Window on the right-hand side of the Test RealTime user interface, right-click a test node and select View Report->Test.
To open these reports using the Test RealTime menu:
- Select the menu item File->Browse Reports.
- Select the Add toolbar button.
- Select Report from the Files of Type dropdown list
- Browse to and then select the .xrd files corresponding to the reports you
would like to view
- Click the Open button
To open these reports from the command line, simply treat each .xrd file as
a parameter to the command line studio .
The report is organized into two main sections, a summary section followed
by a details section.
- Summary - The summary section contains information such as the
report file locations, time of test execution and a summary count of passed
and failed tests.
- Component Test Details - For component tests, this section provides
information about the drivers/services/functions/methods/procedures under
test. For each item, the report contains some or all of the following information,
depending on the source language and the options selected:
- a test section which provides an overview and summarizes whether or
not the test passed
- a variable value section, specifying the acquired value for each variable.
- if code coverage analysis was performed, a code coverage section
- for C++ and Java, additional information is provided regarding contract
(i.e. assertion) validation and verify queries.
- System Test Details - For system tests, the following subsections
are provided:
- an Initialization section, which details the steps involved in opening
a connection to the component under test.
- one or more Scenario sections, which details the actual tests that were
performed. Scenario blocks can be nested.
- a Termination section, which details the steps involved in terminating
connection to the component under test.
The Report Explorer of the Test RealTime user interface can be used to maneuver
through the test report (double-click on any node of interest); the Test Report
menu item and the toolbar let you filter the report in order to only view failures.
Multiple sections of the test report are hyperlinked to the test script; simply
right-click those sections in which the mouse icon resembles a hand.
For detailed
information about viewing test reports see the Rational Test RealTime User
Guide, and refer to the chapter Automated Testing->Component Testing,
in the sections discussing the test report.
For
detailed technical information on working with report files, refer to the Rational
Test RealTime Reference Guide, focusing on the sections discussing the various
test report generators. In particular, see the page Appendices->File Types.
Memory profiling reports for all supported languages-C, C++ and Java-are
stored in XML-based files within the Project folder or the TDP folder, depending
on the language.
To open these reports using the Project Browser tab in the Project Window on
the right-hand side of the Test RealTime user interface, right-click a test
node and select View Report->Memory Profile.
To open these reports using the Test RealTime menu:
- Select the menu item File->Browse Reports.
- Select the Add toolbar button.
- Select Memory Profiling from the Files of Type dropdown list.
- Browse to and then select the files you are interested in viewing.
- Repeat steps 2-4 as needed.
- Click the Open button in the Report Browser window.
To open these reports from the command line, simply treat the one or more files
you are interested in viewing as parameters to the command line studio .
The report is organized into two main sections, a summary section followed
by a details section.
- Summary - A Histogram and textual report section giving the total
number of blocks and bytes allocated and freed, as well as the maximum number
of blocks/bytes allocated at any given time.
- C and C++ Details - Each error and warning is listed, with its
associated call stack. The call stack is listed with the last called function/method
mentioned first; all functions/methods in the call stack are hyperlinked to
source code
- Java Details - A listing is presented of every method that has
allocated objects since the last snapshot. The basic table lists the number
of objects and associated bytes allocated since the last snapshot for each
method, as well as the total number of objects and bytes allocated by each
method and its descendants (i.e. child methods of the parent).
The Report Explorer of the Test RealTime user interface can be used to maneuver
through the test report (double-click on any node of interest); the Test Report
menu item and the toolbar let you filter the report in order to only view failures.
Multiple sections of the test report are hyperlinked to the test script; simply
right-click those sections in which the mouse icon resembles a hand.
For detailed
information about viewing memory profiling reports see the Rational Test
RealTime User Guide, and refer to the chapter Runtime Analysis->Memory
Profiling.
For
detailed technical information on working with report files, refer to the Rational
Test RealTime Reference Guide, focusing on the sections discussing the various
test report generators. In particular, see the page Appendices->File Types.
Performance profiling reports for all supported languages-C, C++ and Java-are
stored in XML-based files within the Project folder or the TDP folder, depending
on the language.
To open these reports using the Project Browser tab in the Project Window on
the right-hand side of the Test RealTime user interface, right-click a test
node and select View Report->Memory Profile.
To open these reports using the Test RealTime menu:
- Select the menu item File->Browse Reports.
- Select the Add toolbar button.
- Select Performance Profiling from the Files of Type dropdown list.
- Browse to and then select the files you are interested in viewing.
- Repeat steps 2-4 as needed.
- Click the Open button in the Report Browser window.
To open these reports from the command line, simply treat the one or more files
you are interested in viewing as parameters to the command line studio .
The report is organized as follows: at the top of each report is a pie chart
detailing each function/method (up to six) that monopolize 5 or more percent
of total execution time. Beneath the pie chart is a sortable table listing each
function/method and various associated statistics. A mouse click on a column
header will sort the overall list by the contents of that column; a mouse click
on any function/method named opens the corresponding source code. No value is
necessarily good or bad - large function execution times, or target function
+ descendant execution times, simply highlights potentially inefficient algorithms
and thus candidates for optimization.
For detailed
information about viewing performance profiling reports see the Rational
Test RealTime User Guide, and refer to the chapter Runtime Analysis->Performance
Profiling.
For
detailed technical information on working with report files, refer to the Rational
Test RealTime Reference Guide, focusing on the sections discussing the various
test report generators. In particular, see the page Appendices->File Types.
Code coverage analysis reports for all supported languages-C, C++, Ada
and Java-are stored in XML-based files within the Project folder or the
TDP folder, depending on the language.
To open these reports using the Project Browser tab in the Project Window on
the right-hand side of the Test RealTime user interface, right-click a test
node and select View Report->Memory Profile.
To open these reports using the Test RealTime menu:
- Select the menu item File->Browse Reports.
- Select the Add toolbar button.
- Select Code Coverage from the Files of Type dropdown list.
- Browse to and then select the files you are interested in viewing.
- Repeat steps 2-4 as needed.
- Click the Open button in the Report Browser window.
To open these reports from the command line, simply treat the one or more files
you are interested in viewing as parameters to the command line studio .
The report is organized into two main components, a Source component and a
Rates component.
- Source - The Source tab reflects two levels of code coverage
information:
- When the Root folder is selected, the Source tab summarizes overall
code coverage using a bar chart.
- When any other node in the Report Window is selected, the Source tab
uses a color-coded display to reflect which code has and has not been
covered. The Source tab displays information related to whichever node
has been selected.
- Rates - The Rates tab breaks down the level of coverage for each
function/method/procedure into percentages.
For detailed
information about viewing Code Coverage reports see the Rational Test RealTime
User Guide, and refer to the chapter Runtime Analysis->Code Coverage.
For
detailed technical information on working with report files, refer to the Rational
Test RealTime Reference Guide, focusing on the sections discussing the various
test report generators. In particular, see the page Appendices->File Types.
Runtime tracing reports for all supported languages-C, C++ and Java-are
stored in XML-based files within the Project folder or the TDP folder, depending
on the language.
To open these reports using the Project Browser tab in the Project Window on
the right-hand side of the Test RealTime user interface, right-click a test
node and select View Report->Memory Profile.
To open these reports using the Test RealTime menu:
- Select the menu item File->Browse Reports.
- Select the Add toolbar button.
- Select Trace Files from the Files of Type dropdown list.
- Browse to and then select the files you are interested in viewing.
- Repeat steps 2-4 as needed.
- Click the Open button in the Report Browser window.
To open these reports from the command line, simply treat the one or more files
you are interested in viewing as parameters to the command line studio .
The runtime tracing report is a sequence diagram of all events that occurred
during the execution of application or test code. This sequence diagram uses
a notation taken from the Unified Modeling Language, thus it can be correctly
referred to as a UML-based sequence diagram. Vertical lines are lifelines; each
lifeline represents a Java object instance. The very first lifeline, represented
by a stick figure, is considered the "world" - that is, the operating
system. Horizontal lines connect one lifeline to another. Green lines are constructor
calls, black lines are method calls, red lines are method returns, blue lines
are destructor calls, orange lines are exceptions. Hovering the mouse over any
method call to see the full text. Every call and call return is time stamped.
Every vertical and horizontal line is hyperlinked to the monitored source code.
The menu item Runtime Trace->Filters can be used to create filters that make
the runtime tracing report more manageable.
For detailed
information about viewing runtime tracing reports see the Rational Test RealTime
User Guide, and refer to the chapter Runtime Analysis->Runtime Tracing
.
For
detailed technical information on working with report files, refer to the Rational
Test RealTime Reference Guide, focusing on the sections discussing the various
test report generators. In particular, see the page Appendices->File Types.
Memory profiling reports for all supported languages-C, C++ and Java-are
stored in XML-based files within the Project folder or the TDP folder, depending
on the language.
To open these reports using the Project Browser tab in the Project Window on
the right-hand side of the Test RealTime user interface, right-click a test
node and select View Report->Memory Profile.
To open these reports using the Test RealTime menu:
- Select the menu item File->Browse Reports.
- Select the Add toolbar button.
- Select Metric from the Files of Type dropdown list.
- Browse to and then select the files you are interested in viewing.
- Repeat steps 2-4 as needed.
- Click the Open button in the Report Browser window.
To open these reports from the command line, simply treat the one or more files
you are interested in viewing as parameters to the command line studio .
The report is organized into two levels, the File view and the Object view:
- File view - When the Root folder is selected, a bar graph of
one of seven Halstead metrics is presented for every file that was selected.
The particular metric that is displayed can be changed using the menu. When
any other node is selected, static metrics specific to that node are presented.
- Object View - When the Root folder is selected, a graph is presented.
This graph compares certain static metrics for each function/method - this
can be changed using the menu. When any other node is selected, static metrics
specific to that node are presented.
For detailed
information about viewing static metrics reports see the Rational Test RealTime
User Guide, and refer to the chapter Runtime Analysis->Static Metrics.
For
detailed technical information on working with report files, refer to the Rational
Test RealTime Reference Guide, focusing on the sections discussing the various
test report generators. In particular, see the page Appendices->File Types.
|