Artifact: Test Suite
|
A package-like artifact used to group collections of tests, both to sequence the execution of those tests and to provide a useful and related set of test log information from which test results can be determined. |
|---|---|
| Role: | Test Designer |
| Optionality/Occurrence: | One or more artifacts. Considered informal and optional in some domains. Where not used, individual tests are executed relatively independently. |
| Templates and Reports: | |
| Examples: | |
| UML Representation: | Not applicable. |
| More Information: |
| Input to Activities: | Output from Activities: |
Purpose
The Test Suite provides a means of managing the complexity of the test implementation. Many system test efforts fail because the team gets lost in the minutia of all of the detailed tests, and subsequently loses control of the test effort. Similar to UML packages, Test Suites provide a hierarchy of encapsulating containers to help manage the test implementation. They provide a means of managing the strategic aspects of the test effort by collecting tests together in related groups that can be reasoned about, planned for, managed, and assessed in a meaningful way.
Brief Outline
Each Test Suite needs to consider various aspects, including the following:
- compatibly and relevance of the individual tests to be executed by the Test Suite, especially in terms of test objective and scope
- points from which the Test Suite can be recovered or resumed if execution is halted
- required configuration settings for the Test Suite of both hardware and software; for example, video-display resolution, resource allocation, environment variables, and so forth
- pre-existing consumables required by the Test Suite, such as populated data sets, receipt printer dockets, and so on
Properties
There are no UML representations for this artifact or its properties.
| Property Name | Brief Description |
|---|---|
| Name | A unique name used to identify this Test Suite |
| Description | A short description of the contents of the Test Suite, typically giving some high-level indication of complexity and scope. |
| Purpose | An explanation of what this Test Suite represents and why it is important. |
| Dependent Test and Evaluation Items | Some form of traceability or dependency mapping to specific elements such as individual requirements that need to be referenced. |
| Preconditions | The starting state that must be achieved prior to executing the Test Suite. |
| Call Sequence Instructions for Tests | The step-by-step instructions for executing the Tests in sequence. |
| Test Suite Observation Points | One or more locations in the instructions where some aspect of the system state will be observed and usually compared with an expected result. |
| Test Suite Control Points | One or more locations in the instructions where some condition or event in the system may occur and needs to be considered in regard to the next Test instruction to be followed. |
| Test Suite Log Points | One or more locations in the instructions where some aspect of the executing Test Suite state is recorded for the purpose of future reference. |
| Postconditions | The resulting state that the system must be left in after the Test Suite has been executed. |
Timing
You can begin identifying candidate Test Suites as early as the Inception phase. Implementation of the Test Suite can typically begin as soon as a test has been identified to be implemented. An approved software build is usually required before it is worthwhile executing a Test Suite to capture measurable test results. For each test cycle, it's useful to execute an initial Test Suite that confirms the stability of the build is adequate to warrant executing additional Test Suites.
Responsibility
The Test Designer role is primarily responsible for this artifact. The details of that responsibility are split into two main areas of concern:
The primary set of responsibilities covers the following design and implementation issues:
- implementing each Test Suite, and managing all subsequent changes to it
- ensuring the Test Suite accurately reflects the test idea being realized
- ensuring the Test Suite is implemented according to defined standards so as to be compatible and maintainable with other Test Suites, and with any Test Scripts it is dependent on.
- ensuring the Test Suite makes reasonably efficient use of the available resources
- developing the Test Suite with a focus on economy of effort, primarily by identifying opportunities for reuse and maintenance simplification
- developing the Test Suite so that it can be used as part of a larger Test Suite
The secondary set of responsibilities covers the following management issues:
- identifying the need for, and outlining the requirements of, each Test Suite
- ensuring the Test Suite encompasses a collection of tests that are useful to validate together, enabling evaluation of useful aspects of the target test items.
Tailoring
This artifact represents a container for organizing arbitrary collections of related tests. This may be realized (implemented) as one or more automated regression Test Suites, but the Test Suite can also be a work plan for the implementation of a group of related manual tests. Note also that Test Suites can be nested hierarchically, therefore one Test Suite may be enclosed within another.
Sometimes these groups of tests will relate directly to a subsystem or other system design element, but other times they'll relate directly to things such as quality dimensions, core "mission critical" functions, requirements compliance, standards adherence, and many others concerns that cut across, or are not directly related to, the internal system elements.
You should consider creating Test Suites that arrange the available Test Scripts-in addition to other Test Suites-in many different combinations: the more variations you have, the more you'll increase coverage and the potential for finding errors. Give thought to a variety of Test Suites that will cover the breadth and depth of the target test items. Remember the corresponding implication that a single Test Script (or Test Suite) may appear in many different Test Suites.
Some test automation tools provide the ability to automatically generate or assemble Test Suites. There are also implementation techniques that enable automated Test Suites to dynamically select all or part of their component Test Scripts for each test cycle run.
