Table of Contents

1. Objectives
2. Scope
3. References
4. Introduction
5. Test Coverage
6. Code Coverage
7. Defect Analysis

   7.1    Defect Density

   7.2    Defect Trend

   7.3    Defect Aging

8. Suggested Actions
9. Diagrams

1. Objectives

This Test Evaluation Report describes the results of the C-Registration Release 1.0 system tests in terms of test coverage (both requirements-based and code-based coverage) and defect analysis (i.e. defect density). These tests were conducted during the C2 Iteration.

2. Scope

This Test Evaluation Report applies to the C-Registration R1.0 Release implemented in the C2 Iteration. The tests conducted are described in the Test Plan [5]. This Evaluation Report is to be used for the following:

• assess the acceptability and appropriateness of the performance behavior(s) of the R1.0 system,
• assess the acceptability of the tests, and
• identify improvements to increase test coverage and / or test quality.

3. References

Applicable references are:

1. Course Registration System Glossary, WyIT406, V2.0, 1999, Wylie College IT.
2. Course Registration System Software Development Plan, WyIT418, V1.0, 1999, Wylie College IT.
3. Course Registration System C2 Iteration Plan, WyIT500, V1.0, 1999, Wylie College IT.
4. Course Registration System C2 Integration Build Plan, WyIT502, V1.0, 1999, Wylie College IT.
5. Course Registration System Test Plan, WyIT501, V1.0, 1999, Wylie College IT.

4. Introduction

The test cases defined in the Test Suite were executed following the test strategy as defined in the Test Plan [5]. The test defects have been logged and any medium, high, or critical priority defects are currently assigned to the owner for fixing.

Test coverage (see Section 5.0 below) in terms of covering the use cases and test requirements defined in the Test Plan [5] was 95% complete. 10 test cases involving operation of the system under a full load were not completed due to bugs in the Load Simulator Software.

Code coverage was measured using Rational Visual PureCoverage and is described in Section 6.0.

Analysis of the defects (as shown in Section 7.0 below) indicates that the majority of found defects tend to be major problems classified as high or critical in severity. The other significant finding was that software components comprising the interface to the Course Catalog System contained a significant number of defects.

5. Test Coverage

All test cases, as defined in the Test Suite, were attempted. 10 tests did not complete due to software errors in the Load Simulator Software. Of the tests cases executed, 15 tests failed.

The test coverage results are as follows:

Ratio Test Cases Performed = 110/120 = 92%

Ratio Test Cases Successful = 95/110 = 87%

The area of tests with the highest failure rate was:

Further detail on test coverage is available using Rational RequisitePro and the Test Case matrix.

6. Code Coverage

Visual PureCoverage was used to measure code coverage of the tests.

Ratio LOC executed = 94,399 / 102,000 = 93%

Approximately, 93% of the code was executed during the testing. This coverage exceeded the target of 90%.

7. Defect Analysis

This section summarizes the results of defect analysis that was generated using Rational ClearQuest. Section 8 recommends actions to address the findings of the defect analysis.

7.1     Defect Density

Data on defect density has been generated using data extracted from ClearQuest reports. Section 9 of this document includes charts that illustrate:

The Defects by Severity Level Chart shows that 26 of the 36 logged defects are classified as high or critical in severity. This number is considered very high and in addition all high and critical defects must be closed in order for the Release to go out.

The Defect Source Chart shows an unusually high percentage of defects are associated with the components (c-abx, c-xxx) which form the interface to the Course Catalog System. In addition, many defects reside within the software components that control installation of the client software.

The Defect Status Chart shows that the defects are promptly being assigned and worked on. Majority of defects have been verified and closed.

In addition, analysis of the critical and high defects has shown that many of these defects are due to poor response times in accessing the Course Catalog System during conditions of heavy load. (Only 50% of the Test Cases verifying performance requirements passed.)

7.2     Defect Trend

Defect trends (i.e. defect counts over time) is shown in the Diagram in Section 9. This trend shows us that the occurrence of defects remains high. If the trend continues, additional iterations may be necessary to find remaining defects in the code.

7.3     Defect Aging

The Defect Aging Chart (see Section 9) illustrates that the time to close some defects is more than 30 days.

The recommended actions are as follows:

1. Continue to devote systems engineering resources to the response time issue involving the Course Catalog System. This is a critical issue as the R1.0 Release cannot be released without meeting the performance requirements.
2. Review the master schedule to see if a fourth iteration can be added to the Construction Phase. The trend in defects over time indicates that many defects remain in the code and an additional test cycle is recommended.
3. Components with a high defect rate should be inspected prior to resubmitting to a build. This includes c-abx and c-xxx.
4. The high rate of Critical and High severity defects may indicate that the design was incomplete and not properly reviewed. Plan additional design reviews for the R2.0 Release.
5. Fix the problems with the Load Simulator Software and re-run the associated test cases.
6. Investigate defect aging. Why are a number of defects taking more than 30 days to close?