When there is a discussion around PERT, the Central Limit Theorem, estimating durations, efforts, and costs, or any probabilistic or statistical aspect of a project, there is usually a fundamental flaw in the thought process.

When we estimate, we are ALWAYS subject to anchoring and adjustments. It is built into our nature. No matter how well you rationalize the processes for capturing estimates, you cannot avoid anchoring and adjusting. Recognition and correction for anchoring and adjusting is the only way out. The estimates are still biased, they are now recognized as biased.

The core of the Kahneman and Tversky thesis is the contention that people frequently form estimates by starting with a given, easily available reference value — which could be arbitrary — and adjusting from that value. An estimate, therefore, would be "anchored" to that value.

Most estimators are in fact "irrational beings" in the sense of risk taking, bounding of outcomes, and their deviations for the "norm," what ever that "norm" is. This is the basis of "models," calibrated models.

This takes us back to core issue - How do we construct a credible Performance Measurement Baseline? If you're not using estimating models, built and validated by the subject matter experts, and calibrated against past performance, then the credibility question is in question.

"volatility per se, be it related to weather, timing of the morning newspaper, is simply a benign statistical probability factor that tells us nothing about risk until it is coupled with a consequence," - Robert H. Jeffrey, "A New Paradigm for Risk," Journal of Portfolio Management, Volume 11, Number 1 (Fall), pp. 33-40.

When we speak about capturing ranges of duration or cost, or speak about compliance with technical performance measures and do not speak about the consequences, then we're pretty much wasting our time with Risk Management.

Here's The Core Problem(s)

• When capturing the estimates using what every method you choose - I'd recommend NOT asking the estimator the Hi, Most Likely, and Lo for lots of reasons discussed else where - more information is needed than just the numbers.
• After eliciting the ranges of values, the probability distributions can then be used to drive a Monte Carlo simulator. This approach produces a Credible forecast of the probability of completing On or Before a Date or having the project cost A Value or Less.
• When static 3 point estimates are used, there can be up to a 27% unfavorable (under estimating) of the completion date. So don't use 3 points and don't use PERT.
• There is a whole cottage industry on why the PERT formulas are bogus and the problems with them. Here's a start "Why PERT Has Problems."

For places to look for PERT background start with:

[1] "Quantitative Risk Analysis for Project Management,: A Critical Review," Lionel Galway, Rand Working Paper, WR-112-RC, February 2004.
[2] "The Polaris System Development: Bureaucratic and Programmatic Success in Government," Harvey M. Sapolsky, Harvard University Press, 1972.
[3] "PERT Completion Times Revisited," Ted Williams, University of Michigan-Flint, School of Management, Working Paper 2005-02, September 2005.
[4] "Hidden Assumptions in Project Management Tools," Dr. Eva Reginier, DRMI Newsletter, January 10, 2006, Naval Postgraduate School, Monterey CA.
[5] "Activity Completion Times in PERT and Scheduling Network Simulation," Dr. Eva Reginier, DRMI Newsletter, April 8, 2005, Naval Postgraduate School, Monterrey CA.

There are unlimited opinions floating around about the role of management. Many come from voices that would have management be removed from the process - the extreme agile point of view.

For those looking to understand how management has and is evolving in the "post industrial" age, here's a nice article A NEW ROLE FOR MANAGEMENT IN TODAY’S POST-INDUSTRIAL ORGANIZATION

The Ivey Business Journal is free Business Journal from the Ivey School of Business, University of Western Ontario.

Knowing where you are most likely to fail can help set the stage for reducing the chances of failure.

- Bilal M. Ayyub, Peter G. Prassinos, and John Etherton

PMI has moved to Virtual Communities, away for the standalone organizations of the past. What ever their reasoning, it doesn't matter. I voted against the VC move for College of Performance Management, but we went there anyway.

Now that VC's are starting, I', signing up for their Blogs.

This process is of course broken and lame. No Goggle Reader link, when you navigate to the RSS feed page, and copy the URL to Google Reader it can't find a "read" feed, but provides a "watch" process. This is new for Google Reader and applicable to those sites that are clueless about RSS and Blog Feeds.

Experience with the VC To date - not impressed.

I had the pleasure of speaking at the Fort Worth Texas PMI Chapter meeting and two sessions of the PMI Symposium this June.

Here's the material

Fort Worth Chapter Meeting

PMI chapter meeting (v4)

Symposium: Immutable Principles of Project Management

Immutable principles of project management (fw pmi)(v4)

Symposium: Establishing the Performance Measurement Baseline

Establishing the performance measurement baseline (pmi fort worth)(v4)

Ben Snyder has several posts on PM Hut about the demise of the "Iron Triangle" in PMBOK® Version. For some reason PMI decided to remove this concept, just when those of us in the defense and space business are starting to use the phrase "Ponzi Scheme."

Just as a reminder, the three invariant variables in any project are, Cost, Schedule, and Technical Performance. Technical Performance includes the quality measures from the previous version of PMBOK®. Technical Performance Measures describe the behavior of the product or service in units meaningful to the buyer.

The relationship between these three variables is coupled, in likely non-linear ways, but they are coupled and this coupling is inseparable. There have been those that suggest this coupling can be broken. It can't.

Just so we don't forget - since PMI removed the reminder - making trades between Cost, Schedule, and Technical Performance and NOT expecting unfavorable impacts is a Ponzi Scheme. Treat anyone suggesting those trades can be made with no impact in the same way you'd treat Charles. Run Away.

There are no points for predicting rain, only building lifeboats

-Warren Buffet’s Noah Principle 

When we hear about the "principles" of project management, good principles for sure, we need to ask the next question how can we cause actionable outcomes

When someone tells me they have a schedule, they have a budget, they've got a set of requirements. One starting response is:

Are these schedules, budgets, and requirements credible?

What are the units of measure of credible? How would we recognize credible if it walked in the door?

Let's Look At The Attributes of a Credible Schedule

The National Defense Industry Association (NDIA) has quarterly meetings of the Industrial Committee for Program Management (ICPM). These meetings are a gathering of the industry and government program management community.

During the February 2010 meetings there was a presentation of the Program Planning and Scheduling Subcommittee (PPSS) Initiative, by Ms Lil Vayhinger and Ms Rebecca Davies. One slide in a presentation listed essential elements of the "generally accepted" scheduling principles. These are:

A valid schedule provides reasonable and credible information based on realistic logic, durations, and dates. This information is:

• Complete: The schedule captures the entire discrete, authorized project effort from start through completion.

Schedules represent authorized discrete effort for the entire contract, with essential subcontracted or other external work or milestones integrated yet distinguishable from internal work.

• Traceable: The schedule logic is horizontally and vertically integrated with cross-references to key documents & tools. 

Schedules reflect realistic & meaningful network logic that horizontally and vertically integrates the likely sequence for program execution. Schedules are coded to relate tasks or milestones to source or dependent documents, tools, and responsible organizations.

• Transparent: The schedule provides visibility to assure it is complete, traceable, has documented assumptions, and provides full disclosure of program status and forecast. 

Schedules provide full disclosure of program status and forecast and include documented ground rules, assumptions, and methods for building and maintaining schedules. Documentation includes steps for analyzing the critical paths, incorporating risks and  opportunities, and generating schedule health and performance metrics.

• Statused: The schedule has accurate progress through the status date. 

Schedules reflect consistent and regular updates of completed work, interim progress, achievable remaining durations relative to the status date, and accurately maintained logic relationships.

• Predictive: The schedule provides meaningful critical paths and accurate forecasts for remaining work through program completion.

Schedules accurately forecast the likely completion dates and impacts to the program baseline plan through valid network logic and achievable task durations from the status date through program completion.

An effective schedule is useful, helps align time-phased resources, and is built and maintained using a controlled process. This  information is:

• Usable: The schedule is an indispensable tool for timely and effective management decisions and  actions. 

Schedules produce meaningful metrics for timely and effective communication and tracking and improving performance, mitigating issues and risks, and capturing opportunities. Schedules are robust and functional to help stakeholders manage different levels, groupings, or areas as needed. Schedules are developed and maintained at a size, level, and complexity such that they are timely and enable effective decision-making.

• Resourced: The schedule aligns with actual and projected resource availability.

Resources align with the schedule baseline & forecast to enable stakeholders to view & assess the time-phased labor & other costs required to achieve project baseline & forecast targets. Each program is unique and uses varying techniques to load, baseline, & maintain the time-phased resources at levels that are practical & produce meaningful and accurate projections. When resource-loaded schedules are used they enable flexible updates to resource requirements as conditions change. Whether or not resource-loaded schedules are used, cost & schedule data are integrated for internal & external reporting.

• Controlled: The schedule is built, baselined, and maintained using a stable, repeatable, & documented process. 

Schedules are baselined and maintained using a rigorous, stable, and repeatable process. Schedule additions, deletions, and updates conform to this process and results in valid and accurate results for sound schedule configuration control and maintenance.

These attributes are the basis of a credible schedule and the costs associated with the work delivered by the work elements of the schedule. They are independent of the domain or context of the project. They are independent of the size of the project or program. They are the basis of a "credible" schedule and therefore a "credible" project.

If you have a project that does not have these attributes, you're likely in trouble and you may not know it.

I was surfing some presentations looking for good layout ideas, when I came across a Jazz - A Metaphor for Agile Management. I'm not a real jazz fan, but I do enjoy certain artist. Sitting in the Chautauqua,  performance hall last week for the Brahmas 1st Piano Concerto and thinking about Jazz as a management metaphor, I'm struck by several things:

• How many players can you have in the performance before you need a conductor?
• If I want to perform the 1st Piano Concerto, can it be done without a conductor, individual section music?
• Of course the pianist, Peter Serkin, played without  music, but he "knows" the piece inside outside. The orchestra struggled a bit since they had only rehearsed for two days.
• When you listen to the penultimate performance, I would guess it was rehearsed many dozens of times, Ashkenazy, and Haitink studied the piece and the orchestration for months if not years, and were both at the peak of their performance capabilities before laying down the tracks. 

Finally:

• When the metaphor is applied to software development, are there people (the customer) willing to pay others (the developers) to improvise in the way some Jazz groups do?
• At what point does "spending other peoples money" need oversight in the same way the orchestra needs a conductor.
• What are the limits to the size and complexity of the music or the software that requires external oversight - project management or a conductor.
• What is the customer doesn't like Jazz?

So when does Jazz serve the needs of the customer? When does Jazz run out of capabilities to produce a credible performance? When is Jazz an inapproprite approach to the performance of the music? Why do we think Jazz is all that is needed?

The discussion of estimating - cost and schedule - has produced references usable in a variety of domains. As a background here's the guidance in the domain I work.

 

This "mandates" an Estimate At Completion with a Best Case, Worst Case, Most Likely. As well a Schedule Risk Analysis (SRA) is mandated.

And just for clarification for those favoring "hand built" 3 point estimates, the 3 points estimates stated in DID-81560 above are generated using the "ordinal range" method. The outcome from this processes are the percent ranges - upper and lower - of the probability distribution (Triangle) used for the Monte Carlo tool (Risk+ and @Risk for Project). So we have the three points, but they were not gathered by asking the engineers what they should be. The Systems Engineering, Control Account Managers, and Technical Leads, plus historical data, plus some statistical analysis produced the ordinal ranges for 5 classes of programmatic risk. That is then applied to each class of work, and then loaded into the baselined Integrated Master Schedule.

The point of all this is that producing credible estimates is part of the culture of the work we do. It's still just as hard, just as complex, and just as annoying as any other domain. But doing it, and doing it right is what is expected. It's part of being credible. So for those gathering variance ranges by hand, if the credibility - statistical confidence, auto-correlation, assessment of all the biases and influences discussed in The Psychology of Risk and Sources of Estimating Error can be addressed, then you're on the path to credibility.

Eisenhower had to deal with as

"fractious and dysfunctional a group of egomaniacs as any war had ever seen."

Ike's greatest achievement was keeping the allied command focused on killing the Nazi's not each other.

- From a speech delivered by Secretary of Defense, Robert M. Gates, Abilene, KS, Saturday May 08, 2010.

Mike Clayton pointed me to a post on the The Psychology of Risk. The book in his post Risk: The Science and Politics of Fear, looks like the next reading assignment after our book club finishes Against The Gods.

Mike's post lists four biases for developed by Tversky, Kahnemann, and Slovic.

This book, Mike's post, and the materials in the previous post on Sources of Estimating Error, should put an end to the naive and simple minded approach to gathering estimates on projects.

Please, please, please do not apply three point estimates to anything without first having read the materials referenced here and the Tversky and Kahnemann materials. You will simply be setting the stage for disappointment all around.

Starting with Tversky & Kahnemann (1974), "Judgment under Uncertainty: heuristics and Biases," there are sources of error that should lay the foundation for abandoning the simple and many times naive 3 point estimate approach to work duration and effort.

• Representativeness - occurs where probabilities are evaluated by the degree to which A is representative of B.

The last time we did this, we say durations that looked like this. Or, We've asked the guys down Florida and they think we should be able to get this done in 1,400 hours, that what it took them the last time they did it.

• Insensitivity to prior probability of outcomes - we ignore the past statistical behavior of our work efforts. This is ignorance of Bayesian statistics.

I know they took 1,400 hours, but we've learned a lot from their mistakes, so we can do it for less. We don't really have evidence, but our gut feel is they were not very good at their job and took too long.

• Insensitivity to sample size - this is a very common error, where the estimator makes a forecast independent of the sample size.

My students - all 23 of them - answered a survey I wrote about a behavior of problems with Earned Value, and they came up with answers I'll apply to a broad set of situations. I sampled the people I know and they came up some interested statements of what works and doesn't work. We hand picked 12 managers in our firm, gave them a survey we put together from some outside suggestions and we'll be making changes based on their answers.

• Misconceptions of Chance - the belief that random processes represent the core behaviors of a process, while observing a short sequence of outcomes.

I've seen this happen 3 or 4 times in other places, it's got to be the same thing happening here. It's happening in all kinds of places, so it's got to be the same here. "For every up there is a down," let's just wait and we'll be back on track pretty soon.

• Insensitivity of predictability - the future can be predicted intuitive predictions

I've seen this happen before, it's got to be the cause of what's happening this time too.

• Illusion of Validity - the observations we're seeing are a good fit with what we're expecting.

We interviewed a list of people selected by management, and they support the what management thinks is going wrong here. Management has a target budget and schedule in mind, and when we asked the developers - after hearing the management numbers - they came up with about the same numbers.

• Misconceptions of Regression - all those random processes add up to a average we can live with.

When we get all the estimates in, we can find the average variance and use that to forecast the work we're going to be given in the future. This concept is actually mis-represented in a government cost estimating guidebook. Adding a sample set of distributions results in a "normal" distribution.

• Biases due to the effectiveness of a search set -when asked to recall some data or event, the one they come up with is what is most familiar to them.

We met some guys in the cafeteria and they had lots to say about how we should estimate our costs

• Biases due to the retrievability of instances - the data I have at hand is the data I used for my forecast.

We had all the data from the projects they did in Dallas, and used that as our sample data for foresting our performance.

• Adjustment and Anchoring - the estimate is based on the starting value and is then adjusted to get the final answer.

Management gave us a starting point for the cost and schedule, so we need to improve that estimate.

• Biases in the evaluation of conjunctive and disjunctive event - in conjunctive events probabilities are over estimated.

Our rule of thumb has served us well in the past - turns up to be biased

• Anchoring in the assessment of subjective probability distribution - confidence intervals are over confident.

Let's not broaden the variances too much on those cost estimates, it just doesn't feel like right in this situation.

These topics are from Micheal Axelsen's review of of Tversky & Kahnemann 1974 paper, "Judgment under uncertainty: heuristics and biases."

These topics will hopefully put an end to the simple minded 3-point estimates extracted from the people tasked with doing the work. They certainty have a contribution to the estimating process. But serious problems arise when you take those numbers and start making management decisions. In exactly the same way you create serious problems when you take management directive and start makes estimates "anchored" on the bounds of cost or schedule they what to have you produce to.

The standard approach to project taskl duration estimating is the PMBOK® 3 point estimate. Josh has provided several approaches to the discussion of the issues.The top two issues are:

• The anchoring and adjusting error occur when humans make estimates in the presence of uncertainty.
• Failure to understand the underlying statistical distributions produce unfavorable confidence in the estimates

Stochastic process models of task durations appear in other domains - shop floor scheduling, interval estimates, and constrained resource management - are examples. Bayesian networks are another. These paradigms offer significant improvement in the confidence of the estimates needed for credible schedule and cost estimates for projects in all domains.

The issues with the traditional - and many times naive - approaches include:

• Unrealistic assumptions of probabilistic independence.
• Failure to take into account the "estimating in the presence of uncertainty" described in the anchoring and adjustment research.
• The stochastic nature of all project work, that is ignored in non-statistical estimating processes.
• Failure to assess the probabilistic critical path

It is Time to Move Forward

For a project to be successful it must balance schedule, cost, and the technical performance measures. The influence on these three elements are probabilistic rather than deterministic. The outcomes from these elements and their relationships are a function of these probabilistic distributions.

Because of this, we must move forward in our understanding of the behaviors of the project...

• No more single point estimates.
• No three point estimates without acknowledgment of their limitations.
• No naive definition of the critical path.
• No more ignoring the complexities of schedule and cost.
• No using simple minded process for complex projects

He who can handle the quickest rate of change survives - Col John Boyd USAF (Ret)

Boyd is the author of the Observation, Orientation, Decision, Action loop.

The OODA loop is a model for decision making that combines theoretical constructs from biology, physics, mathematics, and thermodynamics applied to air combat. OODA is applicable is any domain where rapid adaption to an emerging situation is needed for survival.

There are four process areas (from "Performance Based Leadership Preparation I - American University, Pat Barker instructor, Senior Program Manager Certificate Program, US DoD):

1. Take information pushed to you, or information you pull from the environment.
2. Comprehend what that information means
3. Decide what to do
4. Do it

I'm headed to Fort Worth to speak at the PMI Chapter meeting and conduct a workshop. The topics start with the Immutable Principles of Project Management. Many authors have their lists of principles and practices. Here's what has emerged over the decades for me.

The five immutable principles of project management are:

1. Know where you are going by defining “done” at some point in the future. This point may be far in the future – months or years from now. Or closer in the future days or weeks from now.
2. Have some kind of plan to get to where you are going. This plan can be simple or it can be complex. The fidelity of the plan depends on the tolerance for risk by the users of the plan.
3. Understand the resources needed to execute the plan. How much time and money is needed to reach the destination. This can be fixed or it can be variable.
4. Identify the impediments to progress along the way to the destination. Have some means of removing, avoiding, or ignoring these impediments.
5. Have some way to measure your planned progress, not just your progress. Progress to Plan must be measured in units of physical percent complete.

For the programmatic aspects of project management, these are the fundamentals of success. If you don't have these principles put into practice, your project is probably headed to the ditch.

I removed the Lijit search engine for now. It causes IE8 to show a blank and throw a Java script error. Thank you Microsoft. Chrome and FireFox work fine of course.

Josh from PMStudent.com has a post about estimating. There is a discussion of using 3 point estimates.

Some time ago, I came to the conclusion that making 3 point estimates is a bad idea. Instead use confidence intervals for risk ranges and drive the variance from those bands with a Monte Carlo Simulator.

This is a critical topic that goes to the credibility of any estimate no matter the domain.

The starting point is to understand the concept of anchoring and adjustments to the estimate and their impact on false optimism (or pessimism). The paper to start with is "Anchoring and Adjustment in Software Estimation," Aranda and Easterbrook.

When you read this, you may rethink the approach that asks the estimator the Most Likely estimate and Optimistic and Pessimistic estimates.

The original work in this area is from Tversky and Kahneman and their "Prospect Theory." A good place to look for some details (besides their work) is Against The Gods, Peter Bernstein, in Chapter 16, The Failure of Invariance."

It's time to move on from the classic "simple" approach found in places like PMBOK(r) to the processes used in other places that depend on probabilistic and statistical analysis - engineering, finance, medicine, physics.

Here's the binders for the EADS proposal for the KC-X tanker. The award is worth $35B over its period of performance. Photo source, The DEW Line This is probably the largest award in recent years - no matter who wins... Glen B. Alleman