Activity: Assess and Advocate Quality

Examine the most current Test Evaluation Summaries

Begin by examining the Test Evaluation Summaries that the test team has prepared. Compare the evaluation information to the Test Plan for the iteration to understand the summary in the context of the planned work. Raise any ambiguities and concerns with the test team members who prepared the summary and resolve them.

For this step and subsequent steps that deal with gathering information and assessing the software quality, try to obtain a balanced view incorporating both objective and subjective measures. Remember that objective numbers only give part of the picture and need to be supported and explained by the current project "climate". Conversely, don't rely purely on hearsay and subjective speculation about the software quality: look for supporting objective evidence. We recommend you supplement your objective data by discussion with either team leads or where possible individual team members to gather subjective assessments and gauge how much confidence you can place in the objective data.

Examine selected Test Results for additional context

Based on the Test Evaluation Summaries, examine selected Test Results for additional context. Research the results enough to feel confident you understand the important issues that have been identified in the Test Evaluation Summaries.

Also review the data yourself and look for important trends evident in the Test Results data that may have been missed. In general it's more important to identify what the relative trends in the data are indicating rather than looking at absolute numbers. Be on the lookout for indications such as failures in one areas that relate to failures in others.

Examine key Change Requests

We recommend you limit this exercise to the most pressing Issues and associated Change Requests. You'll be able to devote more energy to a smaller number of issues, and they are often more likely to have the most impact on product quality. If you have a longer list of key issues, we recommend you devote appropriate effort to them based on their relative priority: don't waste your resources by championing the least significant issue. However, note that a significant number of outstanding lower-priority Change Requests can make as significant a statement about the products quality as a handful of high-priority changes. Try to group lower-priority Change Requests into logical aspects of quality based on the quality risk the represent. This will help you articulate and advocate their combined effect on quality more clearly.

Identify important trends evident in the general Change Request data. In general it's more important to identify what the relative trends in the data are indicating rather than looking at absolute numbers. Look for positive signs such as a steady, continuous rate of defect resolution, or a gradual ongoing increase and subsequent decrease in resolution rate over time. Be on the lookout for sharp peaks and troughs in resolution rate that indicate the development team may be encountering process, environmental, political or other problems that are reducing their productivity.

Note: you may also want to take the opportunity improve the clarity of the associated Change Requests, eliminating ambiguity and emotive language and reasoning. If you make these changes yourself, discuss your improvements with the individuals who originally created these artifacts so that they can understand why the improvements are important.

Identify quality gaps and assess the associated impact and risk

Identify each gap in quality and assess the associated impact and risk of each Issue that creates the gap. Consider mitigation and contingency strategies, formulate your initial ideas for these and discuss them with other team members.

Consider that "perfect" quality is arguably a somewhat mythical concept. Be careful to use a realistic and attainable "quality bar" when assessing quality and identifying quality gaps. See Concept: Product Quality

Identify the essential actions to address quality gaps

For each key gap in quality, consider potential mitigation and contingency strategies. Formulate your initial ideas for these and discuss them informally with other team members to gain greater insight and validate your thoughts. In the case of solutions, it's good to have options: they help you weigh up the tradeoffs and take the best solution for the given context.

Work toward a set of useful candidate solutions and suggestions that will aid the project team in suitably addressing each quality gap. It's important for you to do this so that the test effort is recognized as contributing helpfully to problem resolution: not simply reporting problems. This is an important aspect of advocating the value of the test team and gaining respect and cooperation from other team members.

Identify and engage with champions on major issues

It's no fun to back a lost cause. It's usually a more effective approach to identify solutions to problems that the project team are more likely to get behind, accept and commit to achieving. Keep a close relationship with key decision makers, and consider starting by making these key issues visibly informally through one-on-one discussion. Often that's a great way to win support, and find achievable solutions.

Sometimes you don't have any choice but to back a solution that is unpopular with the development team. Where this looks likely, it's even more important to know who you can expect support form and find ways to sell the solution that present it's value as clearly as possible or explain clearly the worse situation that will arise by not resolving the problem.

Negotiate work priority

This is the crux of quality advocacy: being able to negotiate a suitable solution that both appeases the development team and does not significantly reduce the quality of the product. Remember that in most cases the test team primarily an advisor about quality, so you must be careful not to demand a given resolution being taken.

However, it's important that the test team does a good job as an advocate for quality and that includes sometimes being the bearer of news the project team would really not like to hear. This is where good test teams provide the development effort with as much insight into the problem, it's potential solutions and as understanding as the tradeoff's for each choice as possible. You should act to some extent as an agent for the eventual customers of the product and help negotiate solutions that will be in their best interest.

Monitor work progress

Sometimes Defects and other Change Requests get lost in a sea of ongoing basic product development and feature expansion. This is partly because it's more attractive for developers to work on "new stuff" than it is to fix old and buggy code, and partly because business-value can be more obviously placed on adding something new than fixing something broken. As quality advocates, the test team needs to help the project see important defect fixes through to completion.

Successful software teams find a good balance between incremental quality improvement through defect resolution and incremental feature expansion. The test team can assist the project team by finding ways to encourage and support incremental quality improvement rather than taking less helpful and more adversarial "quality police" role.

Confirm appropriate resolution of key Issues

Whatever solution the development team decide on to resolve a quality issue, the resolution should ultimately improve quality. Be sure that you take time to assess the improvement in quality brought by a given resolution and that it both addresses the original issue and does not adversely impact quality in other ways.

For solutions that carry some level of risk themselves, it may be useful to conduct some testing of an early release candidate before too much time and effort is devoted to following the resolution to it's conclusion.

Evaluate and verify your results

Now that you have completed the work, it is beneficial to verify that the work was of sufficient value, and that you did not simply consume vast quantities of paper. You should evaluate whether your work is of appropriate quality, and that it is complete enough to be useful to those team members who will make subsequent use of it as input to their work. Where possible, use the checklists provided in RUP to verify that quality and completeness are "good enough".

Have the people performing the downstream activities that rely on your work as input take part in reviewing your interim work. Do this while you still have time available to take action to address their concerns. You should also evaluate your work against the key input artifacts to make sure you have represented them accurately and sufficiently. It may be useful to have the author of the input artifact review your work on this basis.

Try to remember that that RUP is an iterative process and that in many cases artifacts evolve over time. As such, it is not usually necessary-and is often counterproductive-to fully-form an artifact that will only be partially used or will not be used at all in immediately subsequent work. This is because there is a high probability that the situation surrounding the artifact will change-and the assumptions made when the artifact was created proven incorrect-before the artifact is used, resulting in wasted effort and costly rework. Also avoid the trap of spending too many cycles on presentation to the detriment of content value. In project environments where presentation has importance and economic value as a project deliverable, you might want to consider using an administrative resource to perform presentation tasks.

Rational Unified Process