Activity: Assess and Improve Test Effort

Capture work status

There are different ways to approach this step, and much of the approach will depend on your project culture. Where available, gather and collate progress reports prepared by individual team members or sub-teams. Project time sheets are another possible source to consider. Where project scheduling systems such as Microsoft Project are actively used and updated with actual progress this provides another useful information source. Where available and actively used, you might also derive objective status or progress metrics from configuration and change management systems.

For this step and subsequent steps that deal with gathering information and assessing the test effort, try to obtain a balanced view incorporating both objective and subjective measures. Remember that objective numbers only give part of the picture and need to be supported and explained by the current project "climate". Conversely, don't rely purely on hearsay and subjective speculation about the test effort: look for supporting objective evidence. We recommend you supplement your objective data by discussion with either team leads or where possible individual team members to gather subjective assessments and gauge how much confidence you can place in the objective data.

Gather test effort productivity and effectiveness metrics

Investigate how much effort has been spent on the identification, definition, design, implementation and execution of tests. Keep an eye out for signs of excessive effort being devoted to one aspect of the test effort to the detriment of others. Look also for areas where effort may be unproductive or not showing sufficient benefit for the level of effort being expended.

Look also at the effectiveness of the testing. Look for data that supports your initial observations of effectiveness. Consider aspects such as defect discovery rate, defect severity counts, duplicate defect statistics, and defects detected as test escapes.

Gather Change Request distribution, trend and aging metrics

Identify important trends evident in the Change Request data. In general it's less important for this activity to spend time analyzing data volumes and more important to identify what the relative data trends are indicating. Look for positive signs such as a steady, continuous rate of defect discovery, or a light ongoing increase or decrease in discovery rate over time. Be on the lookout for sharp peaks and troughs in discovery rate that indicate the test team may be encountering process, environmental, political or other problems that are reducing their productivity.

Look at trends in defects closures. Look for significant increases of closures by development staff as "not reproducible"; identify cases where this is a result of insufficient analysis of the failure having been performed by the test team and quantify the extent of this problem. Look at trends in defects being closed by development staff as "functioning as designed"; identify cases where this is a result of insufficient analysis of the specification having been performed by the test team and quantify the extent of this problem. Be careful to confirm these indications are not false and due instead to overworked developers triaging their workload. Some analysis should also be done of defect verification trends as fixes to defects are released to the test team in subsequent builds: look out for trends that indicate defects awaiting verification by the test team are aging or growing to an unmanageable number.

Look for other trends that indicate problems. Look at the the way in which defects and other change requests have been recorded or managed by the test team: ambiguous and insufficient information on a change request is difficult and frustrating for a developer to take action on. The team should take care to monitor that the quality of the information recorded against defects remains-on average-relatively high. Take the opportunity to improve the clarity of the associated Change Requests, eliminating ambiguity and emotive language and reasoning. Work together with the individuals who created these artifacts to ensure the essence of the problem is clearly stated and encourage them to find factual and accurate ways to approach discussing the Issues.

Also look out for imbalances in defect distribution on a number of different dimensions. Look for functional areas of the application or the specification that have low defect counts raised against them: this may indicate an exposure that insufficient testing has been undertaken in that functional area. Look also at distribution by test team member: there may be indications that individual team members are overworked and that productivity is suffering.

Gather traceability, coverage and dependency metrics

Analyze the state of the traceability relationships between the testing assets-Test Ideas, Test Cases, Test Scripts, Test Suites and Change Requests-and the upstream and downstream assets they relate to. Look for signs that indicate the test effort is focused on the correct areas and a useful set of motivations. Look also for negative indications that suggest certain aspects of testing are missing or are no longer of importance: If the requirements or development teams are working on areas not represented by the current test effort, this should raise concerns.

Evaluate metrics and formulate initial assessment

Collate all of the information you have gathered and evaluate it as a collective whole. Remember that each piece of the data gathered only addresses one aspect of the total assessment, and you must formulate your assessment of the test effort based on a balanced and considered view of all data.

Record you initial assessment in a format that will be suitable for the stakeholders to make comments and give feedback on.

Record findings

This activity produces summary status information that is important to the project manager and other roles in the management team. These roles will use the summary findings to make informed decisions about the project.

We recommend your record some aspects of the test effort assessment in a format that allows subsequent assessments to be compared and contrasted with previous ones. This will enable you to analyze the relative trend in test effort improvements over time.

Present assessment and gather feedback

Present your assessment for stakeholders to comment and offer feedback on. The format or method for doing this will differ from project to project: in some cases it will be a series of informal conversations, in another simply a posting on a project intranet web-site, and in others a formal presentation-choose a format that suits your culture.

Even with the best planning and specification documents possible, there will usually be differences between the original expectation and intent of those documents and the resulting end product. This is as true for testing and evaluating software as it is for the software development itself. The value of this step is to take the opportunity to elicit the stakeholders feedback and identify where the careful planning and documentation has not achieved what was originally expected or intended.

Plan and implement improvement initiatives

Based on your analysis and the feedback you've received from various stakeholders, identify opportunities for improvement. Look for ways to make the testing more effective, productive and efficient. This might involve: reassigning staff, including pairing staff to work more effectively or employing specialized contractors; using productivity tools to improve efficiency; finding alternative approaches and techniques that are more productive in terms of finding defects.

In most cases it's better to make small, incremental improvements to the test effort and avoid the risk of derailing the project with large, unsettling changes: In some cases a bigger change is warranted and useful. Use your best judgment to formulate an appropriate approach to improvement and discuss your ideas with other management staff to get their input before committing the team to embrace large changes.

Monitor and support improvement initiatives

For the improvements to be effective, you will need to manage their success. Identify ways that you will be able to monitor improvement initiatives-preferably in advance on their adoption-to assess their effectiveness. Either actively monitor the progress being made in adopting the changes yourself, or appoint someone else on the team to do so.

Most changes meet resistance or problems that must be overcome for them to be ultimately successful. Allow time for and be prepared to quickly address any issues that arise and prevent the initiative from succeeding. Be sensitive to peoples natural reluctance to change and find ways to address their concerns appropriately.

Evaluate and verify your results

Now that you have completed the work, it is beneficial to verify that the work was of sufficient value, and that you did not simply consume vast quantities of paper. You should evaluate whether your work is of appropriate quality, and that it is complete enough to be useful to those team members who will make subsequent use of it as input to their work. Where possible, use the checklists provided in RUP to verify that quality and completeness are "good enough".

Have the people performing the downstream activities that rely on your work as input take part in reviewing your interim work. Do this while you still have time available to take action to address their concerns. You should also evaluate your work against the key input artifacts to make sure you have represented them accurately and sufficiently. It may be useful to have the author of the input artifact review your work on this basis.

Try to remember that that RUP is an iterative process and that in many cases artifacts evolve over time. As such, it is not usually necessary-and is often counterproductive-to fully-form an artifact that will only be partially used or will not be used at all in immediately subsequent work. This is because there is a high probability that the situation surrounding the artifact will change-and the assumptions made when the artifact was created proven incorrect-before the artifact is used, resulting in wasted effort and costly rework. Also avoid the trap of spending too many cycles on presentation to the detriment of content value. In project environments where presentation has importance and economic value as a project deliverable, you might want to consider using an administrative resource to perform presentation tasks.

Rational Unified Process