Guidelines: Estimating Effort Using the Wide-Band Delphi Technique
Contributed to RUP by Karl Wiegers (www.processimpact.com), with permission
from Software Development Magazine. Further edited by Rational Software
Corporation.
Topics
Introduction
This guideline describes a technique that can be used to estimate software
development effort. The Wideband Delphi estimation method can be summarized
as follows:
- Select a team of experts, and provide each with a description of the problem
to be estimated.
- Ask each expert to provide an estimate (often anonymously) of the effort,
including a breakdown of the problem into a list of tasks, and an effort estimate
for each task.
- The experts then collaborate, revising their estimates iteratively, until
a consensus has been reached.
Using the Wideband Delphi method provides several advantages over obtaining
an estimate from a single individual. First, it helps build a complete task
list or work breakdown structure for major activities, because each participant
will think of tasks. The consensus approach helps eliminate bias in estimates
produced by self-proclaimed experts, inexperienced estimators or influential
individuals who have hidden agendas or divergent objectives. People are generally
more committed to estimates they help produce than to those generated by others.
No participant in an estimation activity knows the "right" answer,
and creating multiple estimates acknowledges this uncertainty. Finally, users
of the Delphi approach recognize the value of iteration on any complex activity.
Applying Wideband Delphi
Wideband Delphi can be used to estimate virtually anything-the number of labor
months needed to implement a specific subsystem, the lines of code or number
of classes in an entire product, or the gallons of paint needed to redecorate
Bill Gates' house, or the effort it would take a particular organization to
achieve level two of the Capability Maturity Model.
The Delphi method helps you develop a detailed work breakdown structure, which
provides the foundation for bottom-up effort and schedule or size estimation.
The starting point for a Delphi session could be a Vision document, a more detailed
Requirements specification of the problem being estimated or an initial high-level
architecture description, or a project schedule. The outputs are a more detailed
project activity list; a list of associated quality, process-related and overhead
activities; estimation assumptions; and a set of activities and overall project
estimates, one from each participant.
Figure 1 illustrates the process flow for a Wideband Delphi session. The problem
being estimated is defined and the participants selected during planning. The
kickoff meeting gets all estimators focused on the problem. Each participant
then individually prepares his or her initial activity lists and estimates.
They bring these items to the estimation meeting, during which several estimating
cycles lead to a more comprehensive activity list and a revised set of estimates.
The moderator or project manager then consolidates the assorted estimation information
offline, and the team reviews the estimation results. When some predetermined
exit criteria are satisfied, the session is completed. The resulting range of
estimates is likely to be a more realistic predictor of the future than any
single estimate. Let's look at each of these process steps in turn.
When planning a Wideband Delphi session, the problem is defined
and the participants selected. The kickoff meeting gets all estimators focused
on the problem. Each participant then individually prepares initial activity
lists and estimates. During the estimation meeting, several cycles lead to a
more comprehensive activity list and a revised set of estimates. The information
is then consolidated offline, and the team reviews the estimation results. When
the exit criteria are satisfied, the session is completed.
Planning
A Wideband Delphi session begins with defining and scoping the problem: vision,
use case model, existing system, preliminary architecture. Large problems are
broken down into manageable portions that can be estimated more accurately,
perhaps by different teams. The person who initiated the estimation activity
assembles a problem specification that will give the participants enough information
to produce credible, informed estimates.
The estimation participants include a moderator, who plans and coordinates
the activity, the project manager and two to four other estimators. The moderator
should be informed enough to participate as an estimator but acts as an impartial
facilitator who won't skew the results with his or her own biases or insights.
The participants are selected because they understand the problem or project
and associated estimation issues.
The Kickoff
An initial kickoff meeting of up to an hour gets all participants up to speed
on the estimation problem. The moderator explains Wideband Delphi to team members
who are unfamiliar with it and supplies the other estimators with the problem
specification and any assumptions or project constraints. The moderator strives
to give the estimators enough information to do a good job without unduly influencing
their estimates.
The team reviews the estimation objectives and discusses the problem and any
estimation issues. The participants agree on the estimation units they will
use, such as weeks, labor hours, dollars or lines of code. If the moderator
concludes that all team members are sufficiently knowledgeable to contribute
to the estimation activity, the group is ready to roll. Otherwise, the participants
may need to be briefed more fully on the problem they're estimating, or possibly
replaced by others who can generate more accurate estimates.
To determine whether you're ready to proceed with the Wideband Delphi session,
check your entry criteria-that is, the prerequisites that must be satisfied
for you to proceed with subsequent process steps. Before you dive into the estimation
exercise, ensure that the following conditions are satisfied:
- Appropriate team members have been selected.
- The kickoff meeting has been held.
- The participants have agreed on the estimation goal and units.
- The project manager can participate in the session.
- The estimators have the information they need to participate effectively.
Individual Preparation
Let's assume that you wish to estimate the total amount of work effort (typically
expressed in labor hours) needed to complete a certain project. The estimation
process begins with each participant independently developing an initial list
of the tasks that will have to be completed to reach the stated project goal,
using a form like that shown in Figure 2. Each participant then estimates the
effort each task will consume. Break each task down into activities that are
small enough to estimate accurately. State the activities clearly, because someone
will have to merge all of the participant activity lists into a single composite
list. Total the estimates you produce for each project task, in the agreed-upon
units, to generate your initial overall estimate.
The estimation process begins with each participant independently
using this form to develop an initial list of the tasks that will have to be
completed to reach the stated project goal.
Your estimate should have no relationship to the answer you think the project
manager or other stakeholders want to hear. There's a good chance the estimate
will fall outside the acceptable project bounds of schedule, effort or cost,
a situation that demands negotiation and might lead to scope reduction, schedule
extension or resource adjustments. But don't let outside pressure sway your
best projection of how the project will play out.
In addition to identifying the project tasks, separately record any tasks for
related or supporting activities. Do not forgot to list tasks dealing with management,
configuration management and process-related activities on the first cycle.
Be sure to include rework activities following testing or inspection activities.
Reworking to correct defects is a fact of life, so you should plan for it. If
you're estimating a schedule, also think of any overhead activities that aren't
specific to the project that you might have to build into your planning. These
include meetings, vacation, training, other project assignments and myriad other
things that suck time out of your day.
Since radically different assumptions can lead to wide estimate variations,
record any assumptions you made while preparing your estimates. For example,
if you assumed that you will purchase a specific component library or reuse
one from a previous project, write that down. Another estimator might assume
that the project will develop that library, which will lead to a mismatch between
your two overall estimates.
Keep the following estimation guidelines in mind:
- Assume one person (you) will perform all tasks.
- Assume all tasks will be performed sequentially; don't worry about sequencing
and predecessor tasks at this time.
- Assume that you can devote uninterrupted effort to each task (this may
seem absurdly optimistic, but it simplifies the estimation process).
- In units of calendar time, list any known waiting times you expect to encounter
between tasks. This will help you translate effort estimates into schedule
estimates later on.
Estimation Meeting
The moderator begins the estimation meeting by collecting the participants'
individual estimates and creating a chart such as Figure 3. Each participant's
total project estimate is shown as an X on the "Round 1" line. Each
estimator can see where his or her initial value fits along the spectrum. The
initial estimates probably will cover a frighteningly large range. Just imagine
the different conclusions you might have collected had you asked just one of
the participants for his or her estimate and used that to plan the project.
The moderator begins the estimation meeting by collecting and charting
the participants' individual estimates. Each participant's total project estimate
is shown as an X on the "Round 1" line. The initial estimates probably
will cover a frighteningly large range.
In some organization, the moderator does not identify who created each estimate;
they feel this anonymity is an important aspect of the Delphi technique. Anonymity
prevents an outspoken colleague from intimidating the other participants into
seeing things his or her way. It also means team members are less likely to
defer to the most respected participant's judgment when their own analyses lead
to different conclusions. But this is not a must.
Each estimator reads his initial task list, identifying any assumptions made
and raising any questions or issues, without revealing which estimate was his.
Each participant will have listed different tasks that need to be performed.
Combining these individual task lists leads to a more complete list than any
single estimator is likely to produce. This approach will work for up to several
dozen individual tasks. If you have more tasks than that, they might be too
detailed. You may want to break the problem into several subproblems and estimate
them individually.
During this initial discussion, the team members also talk about their assumptions,
estimation issues and questions they have about the problem. As a result, the
team will begin to converge on a shared set of assumptions and a common task
list. Retain this final task list to use as a starting point the next time you
must estimate a similar project.
After this initial discussion, all participants modify their estimates concurrently
(and silently) in the meeting room. They might revise their task lists based
on the information shared during the discussion, and they'll adjust individual
task estimates based on their new understanding of the task scope or changed
assumptions. All estimators can add new tasks to their forms and note any changes
they wish to make to their initial task estimates. The net change for all tasks
equals the change in that participant's overall project estimate.
The moderator collects the revised overall estimates and plots them on the
same chart, on the "Round 2" line. I've done this on a whiteboard
for easy visibility. As Figure 4 illustrates, the second round might lead to
a narrower distribution of estimates centered around a higher mean than the
mean of the Round 1 values. Additional rounds should further narrow the distribution.
The cycle of revising the task list, discussing issues and assumptions and preparing
new estimates continues until:
- you have completed four rounds;
- the estimates have converged to an acceptably narrow range (defined in
advance);
- the allotted estimation meeting time (typically two hours) is over; or
- all participants are unwilling to alter their latest estimates.
After discussion of the initial estimates, all participants modify their
estimates. The moderator collects the revised overall estimates and plots
them on the same chart, on the "Round 2" line. These later rounds
might lead to a narrower distribution of estimates centered around a higher
mean than the mean of the Round 1 values.
The moderator keeps the group on track, time-boxing discussions to 15 or 20
minutes to avoid endless rambling. The moderator should follow effective meeting
facilitation practices, such as starting and ending on time, encouraging all
participants to contribute and maintaining an impartial and non-judgmental environment.
While preserving the anonymity of individual estimates is important for the
first couple of rounds, the team members might agree at some point to put all
their cards on the table and reach closure through an open discussion. This
gives them a chance to discuss tasks for which their estimates vary substantially.
Otherwise, though, the moderator should not identify the individual who produced
each final estimate until the session is completed.
Assembling Tasks
The work isn't done when the estimation meeting concludes. Either the moderator
or the project manager assembles the project tasks and their individual estimates
into a single master task list. This person also merges the individual lists
of assumptions, quality- and process-related activities, overhead tasks and
wait times.
The merging process involves removing duplicate tasks and reaching some reasonable
resolution of different estimates for individual tasks. "Reasonable"
doesn't mean replacing the team's estimates with values the project manager
prefers. Large estimate differences for apparently similar tasks might indicate
that estimators interpreted that activity in different ways. For example, two
people might both have an activity called "implement a class." However,
one estimator might have included unit testing and code review in the task,
while the other meant just the coding effort. All estimators should define their
activities clearly to minimize confusion during this merging step. The merging
step should retain the estimate range for each task, but if one estimator's
task estimate was wildly different from that of the other estimators, understand
it and then perhaps discard or modify it.
Review Results
In the final step, the estimation team reviews the summarized results and reaches
agreement on the final outcome. The project manager provides the other estimators
with the overall task list, individual estimates, cumulative estimates, assumption
list and any other information. Bring the team back together for a 30- to 60-minute
review meeting to bring closure to the estimation activity. This meeting also
provides an opportunity for the team to contemplate this execution of the Wideband
Delphi process and suggest ways it can be improved for future applications.
The participants should make sure the final task list is as complete as possible.
They might have thought of additional tasks since the estimation meeting, which
could be added to the task list now. Check to see whether tasks that had wildly
different individual estimates have been merged in a sensible way. The ultimate
objective is to produce an estimate range that allows the project manager and
other key stakeholders to proceed with project planning and execution at an
acceptable confidence level.
Completing the Estimation
The estimation process is completed when specified exit criteria are satisfied,
at which point you can declare victory and move on with your life. Typical Wideband
Delphi exit criteria are that:
The overall task list has been assembled.
You have a summarized list of estimating assumptions.
The estimators have reached consensus on how their individual estimates were
synthesized into a single set with an acceptable range.
Now you must decide what to do with the data. You could simply average the
final estimates to come up with a single point estimate, which is what the person
who requested the estimate probably wants to hear. However, a simple average
is likely to be too low, and there's merit in retaining the estimate range.
Estimates are predictions of the future, and the range reflects the inherent
uncertainty of gazing into the crystal ball. You might present three numbers:
the average of the estimates as the planned case, the minimum value as the best
case and the maximum as the worst case. Or you could present the average value
as the nominal expected outcome, plus the maximum-minus-the-average value, and
minus the average-minus-the-minimum value.
Each estimate has a certain probability of coming true, so a set of estimates
forms a probability distribution. In Chapter 6 of A Discipline for Software
Engineering (Addison-Wesley, 1995), Watts Humphrey describes a mathematically
precise way to combine multiple estimates and their uncertainties to generate
an overall estimate with upper and lower prediction intervals. Another sophisticated
approach is to perform a Monte Carlo simulation to generate a probability distribution
of possible estimate outcomes based on the final estimate values.
While the results of a Delphi session might not be what the movers and shakers
want to hear, they can decide whether they want to plan their project at a 10
percent confidence level, a 90 percent confidence level or somewhere in-between.
Be sure to compare the actual project results to your estimates to improve your
future estimating accuracy.
Doing It Again (Iterating)
One nice aspect of this method is that after an initial and rather rough estimate
done for example during inception, the estimates can be refined at each phase
(or even at each iteration). The process can be faster if the same estimators
are available, starting where they left at the previous estimation cycle. More
information about the problem is available, some assumptions have been modified,
an architecture is in place to help break down the effort.
The new estimate may have a narrower range, but is not necessarily within the
range of the previous one: it may be higher, or smaller. If higher, there is
a clear risk signal to the project manager, risk that must be tackled at once.
Wideband Delphi Evaluated
No estimation method is perfect; if it were, it would be called prediction,
not estimation. However, the Wideband Delphi technique incorporates some solid
estimating principles. The team approach acknowledges the value of combining
multiple expert perspectives. The range of estimates produced reflects the variability
intrinsic to the estimation process.
Although it takes time and requires a panel of experienced estimators, Wideband
Delphi removes some of the politics from estimation and filters out extreme
initial values.
|