Introduction

Uncertainty is a fact of organizational life –   managers often have to make decisions based on uncertain or incomplete information. Typically such decisions are based on a mix of intuition, experience and blind guesswork or “gut feel”.  In recent years, probabilistic (or statistical) techniques have entered mainstream organizational practice. These have enabled managers to base their decisions and consequent actions on something more than mere subjective judgement – or so the theory goes.

Much of the statistical analysis in organisational theory and research is based  on the assumption that the variables of interest have a Normal (aka Gaussian) distribution. That is, the probability of a variable taking on a particular value can be reckoned from the familiar  bell-shaped curve.  In a paper entitled Beyond Gaussian averages: redirecting organizational science towards extreme events and power laws, Bill McKelvey and Pierpaolo Andriani, suggest that many (if not most) organizational variables aren’t normally distributed, but are better described by power law or   fat-tailed (aka long-tailed or heavy-tailed) distributions. If correct, this has major consequences for quantitative analysis in many areas of organizational theory and practice. To quote from their paper:

Quantitative management researchers tend to presume Gaussian (normal) distributions with matching statistics – for evidence, study any random sample of their current research. Suppose this premise is mostly wrong. It follows that (1) publication decisions based on Gaussian statistics could be mistaken, and (2) advice to managers could be misguided.

Managers generally assume that their actions will not have extreme outcomes. However, if organisational phenomena exhibit power law behaviour, it is possible that seemingly minor actions could have disproportionate results. It is therefore important to understand how such  extreme outcomes  can come about. This post, based on the aforementioned paper and some of the references therein discusses a couple of general mechanisms via which power laws can  arise in organizational phenomena.

I’ll begin by outlining the main differences between normal and power law distributions, and then present a few social phenomena that display power law behaviour. Following that, I get to my main point – a discussion of general mechanisms that underlie power-law type behaviour in organisational phenomena. I conclude by outlining the implication of power-law phenomena for managerial actions and their (intended) outcomes.

Power laws vs. the Normal distribution

Probabilistic variables that are described by the normal distributions tend to take on values that cluster around the average, with the probability dropping off to zero rapidly on either side of the average. In contrast, for long –tailed distributions, there is a small but significant probability that the variable will take on a value that is very far from the average (what is sometimes called a black swan event).  Long-tailed distributions are often  described by power laws.  In such cases, the probability of variable taking a value is described by a function like   where is called the power law exponent .  A well-known power law distribution in business and marketing theory is the Pareto distribution.  An important characteristic of power law distributions  is that they have infinite variances and unstable means, implying that outliers cannot be ignored and that averages are meaningless.

Power laws in social phenomena

In their paper Mckelvey and Andriani mention a number  of examples of power laws in natural and social phenomena.  Examples of the latter include:

1. The sizes of US firms : the probability that a firm is greater than size N (where N is the number of employees), is inversely proportional to N .
2. The number of criminal acts committed by individuals: the frequency of conviction is a power law function of the ranked number of convictions.
3. Information access on the Web: The access rate of new content on the web decays with time according to a power law.
4. Frequency of family names: Frequency of family names has a power law dependence on family size (number of people with the same family name).

Given the ubiquity of power laws in social phenomena, Mckelvey and Adriani suggest that they may be common in organizational phenomena as well.  If this is so, managerial decisions based on the assumption of normality could be wildly incorrect. In effect, such an assumption treats extreme events as aberrations and ignores them. But extreme events have extreme business implications and hence must be factored in to any sensible analysis.

If power laws are indeed as common as claimed, there must be some common underlying mechanism(s) that give rise to them.  We look at a couple of these in the following sections.

Positive feedback

In a classic paper entitled, The Second Cybernetics: Deviation-Amplifying Mutual Causal Processes, published in 1963, Magoroh Maruyama pointed out that small causes can have disproportionate effects if they are amplified through positive feedback.   Audio feedback is a well known example of this process.  What is, perhaps, less well appreciated is that mutually dependent deviation-amplifying processes can cause qualitative changes in the phenomenon of interest. A classic example is the phenomenon of a run on a bank : as people withdraw money in bulk, the likelihood of bank insolvency increases thus causing more people to make withdrawals. The qualitative change at the end of this positive feedback cycle is, of course, the bank going bust.

Maruyama also draws attention to the fact that the law of causality – that similar causes lead to similar effects – needs to be revised in light of positive feedback effects. To quote from his paper:

A sacred law of causality in the classical philosophy stated that similar conditions produce similar effects. Consequently, dissimilar results were attributed to dissimilar conditions. Many scientific researches were dictated by this philosophy. For example, when a scientist tried to find out why two persons under study were different, he looked for a difference in their environment or in their heredity. It did not occur to him that neither environment nor heredity may be responsible for the difference – He overlooked the possibility that some deviation-amplifying interactional process in their personality and in their environment may have produced the difference.

In the light of the deviation-amplifying mutual causal process, the law of causality is now revised to state that similar conditions may result in dissimilar products. It is important to note that this revision is made without the introduction of indeterminism and probabilism. Deviation-amplifying mutual causal processes are possible even within the deterministic universe, and make the revision of the law of causality even within the determinism. Furthermore, when the deviation-amplifying mutual causal process is combined with indeterminism, here again a revision of a basic law becomes necessary. The revision states:

A small initial deviation, which is within the range of high probability, may develop into a deviation of very low probability or more precisely, into a deviation which is very improbable within the framework of probabilistic unidirectional causality.

The effect of positive feedback can be further amplified if the variable of interest is made up of several interdependent (rather than independent) effects. We’ll look at what this means next.

Interdependence, not independence

Typically we invoke probabilities when we are uncertain about outcomes. As an example from project management, the uncertainty in the duration of a project task can be modeled using a probability distribution.  In this case the probability distribution is a characterization of our uncertainty regarding how long it is going to take to complete the task. Now, the accuracy of one’s predictions depends on whether the probability distribution is a good representation of (the yet to materialize) reality.  Where does the distribution come from? Generally one fits the data to an assumed distribution.  This is an important point: the fit is an assumption – one can fit historical data to any reasonable distribution, but one can never be sure that it is the right one. To get the form of the distribution from first principles one has to understand the mechanism behind the quantity  of interest. To do that one has to first figure out what the quantity depends on .  It is hard to do this for  organisational phenomena  because they depend on several factors.

I’ll explain using an example: what does a  project task duration depend on?  There are several possibilities – developer productivity, technology used, working environment or even the quality of the coffee!  Quite possibly it depends on  all of the above and many more factors. Further still, the variables that affect task duration can depend on each other – i.e. they can be correlated.  An example of correlation is the link between productivity and working environment. Such dependencies are  a key difference between Normal and power law distributions. To quote from the paper:

The difference lies in assumptions about the correlations among events. In a Gaussian distribution the data points are assumed to be independent and additive. Independent events generate normal distributions, which sit at the heart of modern statistics. When causal elements are independent-multiplicative they produce a lognormal distribution (see this paper for several examples drawn from science), which turns into a Pareto distribution as the causal complexity increases. When events are interdependent, normality in distributions is not the norm. Instead Paretian distributions dominate because positive feedback processes leading to extreme events occur more frequently than ‘normal’, bell-shaped Gaussian-based statistics lead us to expect. Further, as tension imposed on the data points increases to the limit, they can shift from independent to interdependent.

So, variables that are made up of many independent causes will be normally distributed whereas those that are made up of many interdependent (or correlated) variables will have a power law distribution, particularly if the variables display a positive feedback effect.  See my posts entitled,  Monte Carlo simulation of multiple project tasks and the effect of task duration correlations on project schedules for illustrations of the effects of interdependence and correlations on variables.

Wrapping up

We’ve looked at a couple of general mechanisms which can give rise to power laws in organisations.  In particular, we’ve seen that power laws may lurk in phenomena that are subject to positive feedback and correlation effects. It is important to note that these effects are quite general, so they can apply to diverse organizational phenomena.  For such phenomena, any analysis based on the assumption of Normal statistics will be flawed.

Most management theories assume  simple cause-effect relationships between managerial actions and macro-level outcomes.  This assumption is flawed because  positive feedback effects can cause  qualitative changes in the phenomena studied. Moreover,  it is often difficult to know with certainty all the factors that affect a macro-level quantity becasues  such quantities are typically composed of  several interdependent factors.  In view of this it’s no surprise that managerial actions sometimes lead to unexpected  extreme consequences.

Interdependence, not independence

Typically we invoke probabilities when we are uncertain about outcomes. As an example from project management, the uncertainty in the duration of a project task can be modeled using a probability distribution.  In this case the probability distribution is a characterization of our uncertainty regarding how long it is going to take to complete the task. Now, the accuracy of one’s predictions depends on whether the probability distribution is a good representation of (the yet to materialize) reality.  But where does the distribution itself come from? Generally one fits the data to an assumed distribution.  This is an important point: the fit is an assumption – one can fit historical data to any reasonable distribution, but one can never be sure that it is the right one. To get the form of the distribution from first principles one has to understand the mechanism behind the quantity  of interest. To do that one has to first figure out what the quantity depends on .  It is hard to do this for  organisational phenomena,  which generally cannot be studied in controlled conditions.

To take a concrete example: what does a  project task duration depend on?  Developer competence? Technology used? Autonomy? Quality of the coffee??  Quite possibly it depends on all of the above. But even further, the variables that make up the quantity of interest can depend on each other – i.e. the can be correlated. This is a key difference between Normal and power law distributions. To quote from the paper:

The difference lies in assumptions about the correlations among events. In a Gaussian distribution the data points are assumed to be independent and additive. Independent events generate normal distributions, which sit at the heart of modern statistics. When causal elements are independent-multiplicative they produce a lognormal distribution (see this paper for examples drawn from science), which turns into a Pareto distribution as the causal complexity increases. When events are interdependent, normality in distributions is not the norm. Instead Paretian distributions dominate because positive feedback processes leading to extreme events occur more frequently than ‘normal’, bell-shaped Gaussian-based statistics lead us to expect. Further, as tension imposed on the data points increases to the limit, they can shift from independent to interdependent.

So, variables that are made up of many independent causes will be normally distributed whereas those that are made up of many interdependent (or correlated) variables will have a power law distribution, particularly if the variables display a positive feedback effect.  See my posts entitled,  Monte Carlo simulation of multiple project tasks and the effect of task duration correlations on project schedules for illustrations of the effects of interdependence and correlations on variables.

Scientific management theories assume a simple cause-effect relationship between managerial actions and macro-level outcomes.  In reality however , it is difficult to know with certainty all the factors that affect a macro-level quantity; it is typically influenced by several interdependent factors.  In view of this it’s no surprise that simplistic prescriptions hawked by management gurus and bestsellers seldom help in fixing organisational problems.

Filed under: Causality, Management, Organizations, Power Laws, Probability, Risk analysis, Statistics

Introduction

Those who run projects often spend a large part of their waking hours working or thinking about work.  It is therefore no surprise that the self images and identities of such individuals are affected (if not defined) by their work roles.  In a sense, their identities are colonized (in the sense of “taken over” or “largely defined”) by the project and the larger, permanent organization which hosts the project. A paper entitled, Who is colonizing whom? Intertwined identities in product development projects,  by Thomas Andersson and Mikael Wickelgren explores this issue via a longitudinal case study that was carried out within new product development teams at Volvo. This post is a summary and review of the paper.

From the title of the paper it is evident that the identity issue is more than just a simple   “the project leader (or team’s) identity is defined by project work“  argument . Indeed, those who lead  product development projects  are themselves involved in creating at least three identities: those of the product, project and organization. Further, as I have pointed out in my post on project management in post-bureaucratic organizations, there are contradictions in the way in which project management operates. On the one hand it is seen as a means to direct innovative efforts (such as product development initiatives); on the other it is an essentially top-down,  bureaucratic means of control. Project teams often operate in such contradictory, tension-filled environments.  Although, most project leaders believe they work on projects by choice, it could be that the  not-so-subtle pressures of expectation and social/ professional norms force the choice upon them.

However, as the authors point out, identity construction isn’t enough to explain why folks are willing to work insane hours; there’s something more going on. Indeed there is research that supports the notion that employees identities are moulded  by organizations to suit organizational ends –  in other words, employees identities are colonized by organisations. Andersson and Wickelgren  suggest that the process of  colonization isn’t as straightforward as it seems because employees often actively seek demanding roles. So, who is colonizing whom? Both parties are complicit in the colonization process.  The main aim of the authors is to describe identity construction processes in project work with a focus on how, via the process  proving their competence in project work, project leaders form their  own self identities.

My review follows the format of the paper: I’ll begin  with an overview of the relevant conceptual and theoretical background and then get into the case study.

Identity construction in project management

Individual identities are defined by how people relate to (professional and social) situations. The process of identity construction is an ongoing process of making sense of situations from a personal perspective. An individual is typically subject to many different identities at an aggregate level – for example the professional identity of a project manager, the identity of a parent etc. These can be thought of as certain norms or ways in which people are expected to behave. Individuals identify with aspects of these roles and  act them out, thus  integrating them into  their own (self) identities. The point is that one’s professional identity is a part of one’s self identity.

The process of identity construction is a discursive one: i.e. it depends on how the situations in which the individual finds himself or herself unfold.  In this connection the authors mention two important concepts, identity work and identity regulation. The first is the ongoing process of identity formation (and reformation) through interaction with the situation at hand (which could be a project, say). The second refers to the norms and rules, or the ways in which people are expected to behave in situations (the explicit norms and rules of project management, for instance). It is clear that identity regulation – by laying down appropriate behaviours – can have a “colonizing effect” on people’s identities. The authors make the point that even identity work can have an implicit colonizing effect – an example would be where project leaders identify with their work to such an extent that they go above and beyond the call of duty.

The increasing projectisation of organisations means that many  individuals spend a large part of their work hours in a project environment. As such it is inevitable that this will affect their self-identity.  In a fascinating paper, Lindgren and Packendorff pointed out that there are certain commonly accepted ways of relating to and making sense of project situations.  For example – a project is seen as demanding higher levels of professionalism and loyalty than “normal” organizational work. Such norms constrain choices of those involved in projects. Once signed up, one has to behave in certain accepted ways. The authors make the point that there are very few empirical studies that look into how people handle such a “projectified reality.”

Several researchers have recognized the paradoxical nature of project work. Project-based management is touted as a means to loosen the hold of bureaucratic management structures. However, project management in practice tends to be riddled with (pointless?) bureaucratic procedures. As another example, projects are seen as a means to accelerate organizational learning – by doing new things under controlled, time-bound conditions. Yet, the reality is that when projects are in progress, organisations are loathe to spend time on capturing knowledge. The focus is always on the immediate deliverables rather than longer term learning. Given this, it is no surprise that individuals too, would rather focus on their next project than reflect on what happened on the previous one.  As the authors put it:

…project managers seem ambivalent about their ‘professional identity’; they both aspire to it and resist it. Because of the lack of opportunities for reflection and learning, project workers often seek higher positions in future projects as their reward, with the result that their careers become a series of endless projects requiring increased responsibility and commitment. Emergency situations and problems that arise owing to these time and resource constraints are resolved by heroic actions that gradually become taken-for granted solutions.

It’s a sad commentary on the profession, but I reckon that we, as practitioners, are at least partly to blame.

Project work-life balance

Projects usually operate under tight budgetary and time constraints. Even if one can find more money to throw at a project, it is often impossible to buy more time. As a result those who work on projects often end up working overtime – “doing whatever it takes” to finish the work. But as the authors point out:

‘Doing whatever it takes’ is a very abstract commitment that is hardly measurable since basically it is a social construction dependent on the project leaders’ sense of duty and the external pressures for heroic actions. The dark side of this commitment means long working hours with the inevitable risk of burnout, stress and work/life balance difficulties, all of which may lead to problems with health, general well-being, and family life. The potential damage is as real for the project workers as it is for their organizations.

But it’s even worse because this destructive type of “heroic behaviour”  is  generally seen as distinguishing committed workers from less committed ones.  The authors claim that this goes beyond specific organisations; it is a consequence of projectisation of society in general.

The case study

The qualitative data presented in the case study was gathered by the authors over two periods: 1998-2001 and 2007. In the intervening period, the authors stayed in touch with those in the organisation (Volvo Car Company), but did not actively gather data. Such a longitudinal study (over a long period) enables researchers to study how attitudes  evolve over time. The methodology of the study is best described in their own words:

We studied projects managers who were jointly running new car projects at Volvo Car Corporation (VCC). In addition to direct observations, we video-taped 100 hours of project management meetings and audio-taped individual interviews with all participating project leaders. Our combination of observations and interviews allowed us to observe the practice and everyday lives of the project leaders and to discuss the observations with the interviewees. Thus we were able to observe the project practice closely. In 2007, we interviewed some people from the initial round of interviews who still worked in new car projects. Using this wealth of empirical material, our focus here is on the multi-layered aspects of identity regulation and identity work, especially in terms of colonization, in the studied setting.

The project leaders enjoyed a high status within the company and were often seen as “company heroes, but only as long as their projects succeeded. This created a tremendous pressure to succeed thus making the heroic approach to project management almost inevitable.

Long hours come with the territory

Working long hours is often seen as a hallmark of a dedicated project manager. Further, in many organisations (such as the one studied by the authors) there was the expectation that project managers would sacrifice their own time for the good of the project. As one project leader explained to the authors:

I work all the time! Weekends, evenings…Before Christmas I picked up my husband at a Christmas party in the evening, and then I went back to work and stayed there until midnight. Still, I met a colleague on Saturday morning and was able to do some more work before we left for Christmas. That is typical in my work. I haven’t time for the simplest things in my private life.

The authors make the point that even if a project leader could finish his/her work  within a normal 40 hour work week, others would still question their commitment if they did not work overtime.  Further, if the project did fail, the failure would almost certainly be attributed to the lack of commitment. Project leaders are thus under pressure to work overtime, whether it is needed or not. This is reflected in another comment by a project leader, about what happens between projects:

The period between projects is tough. It takes time to come down. In the beginning it is hard to accept that working 40 hours a week is not the same as having a half-time job, but that is the feeling. […] I have accepted that there are no interesting jobs where you work 40 hours a week, but I think it might be possible to stop at 60 hours a week…

We see that although unreasonable demands are being made on project leaders, they accede to the demands – or even welcome them. So, who is colonizing whom?

Parallel identity construction

The answer to the question in the previous paragraph is complicated by the fact that the project managers who participated in the study wielded considerable influence over the product they were creating. Indeed, this was the intent of the company. That this is so is illustrated by the remarks of an HR director who was involved in the selection of a project leader:

[the candidate] was, along with his competence regarding car development, chosen because he was an almost perfect customer targeted for the V70….Putting him on the management team of the new car project gave him an opportunity to develop the perfect car that satisfied his lifestyle, and the company got the intended car developed. We also used him as an example of our projected customer in a part of our marketing campaign for the V70.

So, selection for prized positions within the company depended on the technical competence and lifestyles of the candidates. In turn, the chosen ones had an opportunity to stamp their personality (identity?) on the product they created.  Identities of people, projects and products were indeed entwined.

Discussion

The authors note that, once selected, project leaders were given considerable autonomy in running their projects. This included the freedom to make choices that influenced the product. Seen in this light, project leaders could stamp their identity on the products they were involved in creating.  However, there were limits to their independence. After all, project leaders depended on the organisation for their livelihoods. Furthermore, their independence was constrained by organizational rules and norms.

Taking another view, one could view the behaviour of the project leaders as being self imposed in the sense that the long hours they put in could be seen as some kind of an addiction to work. Workaholism is an apt term here, I suppose.    At first, the desire to become project leaders spurred them to work hard and once the position was attained they felt the desire to work harder still, possibly to prove that they were worthy of the trust reposed in them by the organisation.

Taking yet another view, one could say that the project leaders had limited choices in both their private and professional lives. On the one hand, there are professional expectations from the company and on the other, expectations from family and friends. Once the individual chose to become a project leader, the choices on offer in both spheres were limited by the choice they made and their personal priorities. All the project leaders interviewed gave their projects priority over all other aspects of their lives. This isn’t surprising because the selection process ensured it. Nevertheless, it does imply that the project leaders accepted that the organisation formed a major part of their self-identities.

It has to be acknowledged that because of their interest in cars, the project leaders were happy to work insane hours. However, equally, the company consciously exploited this interest to the extent that the project leaders believed the project to be the most important aspect of their lives.

Conclusions

Several researchers have suggested that organisations regulate individual identities in a manner that aligns them with organizational objectives (see this paper by Alvesson and Wilmott, for example). At first sight, the foregoing discussion is a case in point. Yet, to some extent, the project leaders in the study believed they had free will – that they made the choices they did because they wanted to. But in the end, the project (and organisation) “wins”. As the authors put it:

To some extent, the project leaders knew they were subjects of control, colonization, and regulation, and yet they chose this career path with full recognition of the consequences for their work/life balance. Their choice meant accepting long workdays and potential emotional and psychological damage in exchange for professional status, job fulfilment, and high compensation. The colonization had consequently moved beyond organizational controland corporate influence. The project leaders were colonized by the projectified society,a situation which made them aspire to the core constructions of the project management..

I suspect that many project managers – particularly those working on high profile projects within their organisations – will find themselves agreeing with the  authors.  Most of us choose to work on ever more challenging projects to further our professional experience, “make a difference” or even “influence the product”.  Regardless of our motives, we generally believe that we make the choice voluntarily. The question is: is this true?

Filed under: Management, Paper Review, Project Management Tagged: Identity Construction, Work Life Balance

Reasoning by analogy refers to the process of drawing inferences based on similarities between different objects or concepts. For example, the electronic-hydraulic analogy is based on similarities between the flow of water in pipes and the flow of electricity in circuits. Closer home, project teams often use analogies when they estimate tasks based on similar work that they did in the past.   Analogical reasoning is powerful because it enables us to leverage existing knowledge in one area to solve problems in other, possibly unfamiliar, areas.  However, such reasoning can also mislead. This post looks at the problem of false analogies in project estimation.

I’ll begin with a story…

–

Some years ago, I was in a discussion with a client, talking about costs and timelines for an application that the client needed. The application was a sales bonus calculator for front-line sales staff.  The client needed an app that would calculate bonuses for each salesperson (based on some reasonably involved calculations) and display them via a web front-end.  Nothing fancy really,  just a run-of-the-mill corporate web-database application. The discussion was proceeding quite nicely until a manager from the client’s side felt obliged to make a remark based on a false analogy. I can’t recall the conversation word-for-word, but it went something like this:

“It can’t be that hard, he said. ” You guys have built a similar application before, your promotional literature says so”.  He knew from our brochure that we had built a bonus calculator before; problem was he didn’t know the details.

There was a brief silence until my boss said, “Umm…yes we have done a superficially similar project before, but the details were very different from this one.”

“How different can it be?” retorted the manager, “bonuses are based on sales data. You process the data based on rules and display it. Yes, the rules are different, but the concept’s the same. You should be able to do it in half the time you’ve quoted.”

My boss countered the argument politely, but the manager would not let it go. They went back and forth a number of times until the sponsor stepped in and asked my boss to ensure that the manager’s concerns were addressed. The issue was resolved later, after my boss stepped the manager through the application, showing him just how different it was from the one they had requested.

–

The technical manager based  his estimate on a superficial similarity between the app we were building for him and one that we had done earlier. Analogies almost always break down when examined in detail. For example, the electronic-hydraulic analogy mentioned in the first paragraph has several limitations. The same is true when comparing two projects or tasks.

An insidious (and dare I say, more common) occurence of such reasoning is when team members themselves draw false analogies.  This happens when they make seemingly harmless (and often tacit) assumptions regarding similarities between tasks  that are actually dissimilar in ways that are important.  See my post on the reference class problem for a discussion of an estimation technique that is prone to incorrect analogical reasoning.

Estimates based on false analogies are a reflection of poorly understood requirements.   This begs the question: why are  requirements misunderstood when most projects involve countless meetings to discuss scope and requirements? In my opinion this happens because talking about requirements doesn’t mean that everyone understands them in the same way. In fact, in most cases different stakeholders walk away from such meetings with their own version of what needs to be done and how to do it.  The first step towards curing the problem of false analogies is to ensure that all stakeholders have a shared understanding of the requirements. This applies  particularly to those who will create the product and those who will use it.  Dialogue mapping, which I’ve discussed at length in several posts on this blog, offers one way to achieve this shared understanding.

Of course, a deep understanding of  the requirements does not by itself   cure the problem of false analogies.  However, it does make estimators aware of what makes a particular project different from all the other ones they’ve done before. This makes it unlikely that they’ll use a false analogy when making their estimates.

Filed under: Communication, Estimation, Project Management

Introduction

Managers have to make decisions based on an imperfect and incomplete knowledge of future events.  One approach to improving managerial decision-making is to quantify uncertainties using probability.  But what does it mean to assign a numerical probability to an event?  For example, what do we mean when we say that the probability of finishing a particular task in 5 days is 0.75?   How is this number to be interpreted? As it turns out there are several ways of interpreting probabilities.  In this post I’ll look at three of these via an example drawn from project estimation.

Although the question raised above may seem somewhat philosophical, it is actually of great practical importance because of the increasing use of probabilistic techniques (such as Monte Carlo methods) in decision making. Those who advocate the use of these methods generally assume that probabilities are magically “given” and that their interpretation is unambiguous. Of course, neither is true – and hence the importance of clarifying what a numerical probability really means.

The example

Assume there’s a task that needs doing – this may be a project task or some other job that a manager is overseeing. Let’s further assume that we know the task can take anywhere between 2 to 8 days to finish, and that we (magically!) have numerical probabilities associated with completion on each of the days (as shown in the table below). I’ll say a teeny bit more about how these probabilities might be estimated shortly.

Task finishes on Probability Day 2 0.05 Day 3 0.15 Day 4 0.3 Day5 0.25 Day 6 .15 Day 7 .075 Day 8 .025

This table is a simple example of what’s technically called a probability distribution. Distributions express probabilities as a function of some variable. In our case the variable is time.

How are these probabilities obtained? There is no set method to do this but commonly used techniques are:

1. By using historical data for similar tasks.
2. By asking experts in the field.

Estimating probabilities is a hard problem. However, my aim in this article is to discuss what probabilities mean, not how they are obtained. So I’ll take the probabilities mentioned above as given and move on.

The rules of probability

Before we discuss the possible interpretations of probability, it is necessary to mention some of the mathematical properties we expect probabilities to possess. Rather than present these in a formal way, I’ll discuss them in the context of our example.

Here they are:

1. All probabilities listed are numbers that lie between 0 (impossible) and 1 (absolute certainty).
2. It is absolutely certain that the task will finish on one of the listed days. That is, the sum of all probabilities equals 1.
3. It is impossible for the task not to finish on one of the listed days. In other words, the probability of the task finishing on a day not listed in the table is 0.
4. The probability of finishing on any one of many days is given by the sum of the probabilities for all those days.  For example, the probability of finishing on day 2 or day 3 is 0.20 (i.e,  0.05+0.15). This holds because the two events are mutually exclusive – that is, the occurence of one event precludes the occurence of the other. Specifically,  if we finish on day 2 we cannot finish on day 3 (or any other day) and vice-versa.

These statements illustrate the mathematical assumptions (or axioms) of probability. I won’t write them out in their full mathematical splendour, those interested in this should head off to the Wikipedia article on the axioms of probability.

Another useful concept is that of cumulative probability which, in our example, is the probability that the task will be completed by a particular day . For example,  the  probability that the task will be completed by day 5  is 0.75  (the sum of probabilities for days 2 through 5).   In general, the cumulative probability of finishing on any particular day is the sum of probabilities of completion for all days up to and including that day.

Interpretations of probability

With that background out of the way, we can get to the main point of this article which is:

What do these probabilities mean?

We’ll explore this question using the cumulative probability example mentioned above,  and by drawing on a paper by Glen Shafer entitled, What is Probability?

OK, so  what is meant by the statement, “There is a 75% chance that the task will finish in 5 days.” ?

It could mean that:

1. If this task is done many times over, it will be completed within 5 days in 75% of the cases. Following Shafer, we’ll call this the frequency interpretation.
2. It is believed that there is a 75% chance of finishing this task in 5 days. Note that belief can be tested by seeing if the person who holds the belief is willing to place a bet on task completion with odds that are equivalent to the believed probability. Shafer calls this the belief interpretation.
3. Based on a comparison to similar tasks this particular task has a 75% chance of finishing in 5 days.  Shafer refers to this as the support interpretation.

(Aside: The belief and support interpretations involve subjective and objective states of knowledge about the events of interest respectively. These are often referred to as subjective and objective Bayesian interpretations because knowledge about these events can be refined using Bayes Theorem, providing one has relevant data regarding the occurrence of events.)

The interesting thing is that all the above interpretations can be shown to  satisfy the axioms of probability discussed earlier (see Shafer’s paper for details). However, it is clear from the above that each of these interpretations have very different meanings.  We’ll take a closer look at this next.

More about the interpretations and their limitations

The frequency interpretation appears to be the most rational one because it interprets probabilities in terms of results of experiments – I.e.  it interprets probabilities as experimental facts, not beliefs. In Shafer’s words:

According to the frequency interpretation, the probability of an event is the long-run frequency with which the event occurs in a certain experimental setup or in a certain population. This frequency is a fact about the experimental setup or the population, a fact independent of any person’s beliefs.

However, there is a big problem here: it assumes that such an experiment can actually be carried out. This definitely isn’t possible in our example: tasks cannot be repeated in exactly the same way – there will always be differences, however small.

There are other problems with the frequency interpretation. Some of these include:

1. There are questions about whether a sequence of trials will converge to a well-defined probability.
2. What if the event cannot be repeated?
3. How does one decide on what makes up the population of all events. This is sometimes called the reference class problem.

See Shafer’s article for more on these.

The belief interpretation treats probabilities as betting odds. In this interpretation a 75% probability of finishing in 5 days means that we’re willing to put up 75 cents to win a dollar if the task finishes in 5 days (or equivalently 25 cents to win a dollar if it doesn’t).  Note that this says nothing about how the bettor arrives at his or her odds.  These are subjective (personal) beliefs. However, they are experimentally determinable – one can  determine peoples’ subjective odds by finding out how theyactually place bets.

There is a good deal of debate about whether the belief interpretation is normative or descriptive: that is, do the rules of probability tell us what people’s beliefs should be or do they tell us what they actually are. Most people trained in statistics would claim the former – that the rules impose conditions that beliefs should satisfy. In contrast, in management and behavioural science, probabilities based on subjective beliefs are often assumed to describe how the world actually is. However, the wealth of literature on cognitive biases suggests that the people’s actual beliefs, as reflected in their decisions, do not conform to the rules of probability.  The latter observation seems to favour normative option, but arguments can be made in support (or refutation) of either position.

The problem mentioned the previous paragraph is a perfect segue into the support interpretation,  according to which the probability of an event occurring is the degree to which we should believe that it will occur (based on available evidence).  This seems fine  until we realize that evidence can come in many “shapes and sizes.”  For example, compare the statements “the last time we did something similar we finished in 5 days, based on which we reckon there’s a 70-80% chance we’ll finish in 5 days” and “based on historical data for gathered for 50 projects, we believe that we have a 75% chance of finishing in 5 days. “ The two pieces of evidence offer very different levels of support. Therefore, although the support interpretation appears to be more objective than the belief interpretation, it isn’t actually so because it is difficult to determine which evidence one should use.  So, unlike the case of subjective beliefs (where one only has to ask people about their personal odds), it is not straightforward to determine these probabilities empirically.

So we’re left with a situation in which we have three interpretations, each of which address specific aspects of probability but also have major shortcomings.

Is there any way to break the impasse?

A resolution?

Shafer suggests that the three interpretations of probability are best viewed as highlighting different aspects of a single situation: that of an idealized case where we have a sequence of experiments with known probabilities.  Let’s see how this statement (which is essentially the frequency interpretation) can be related to the other two interpretations.

Consider my belief that that the task has a 75% chance of finishing in 5 days. This is analogous to saying that if the task is done several times over, I believe it would finish in 5 days in 75% of the cases.  My belief can be objectively confirmed by testing my willingness to put up 75 cents to win a dollar if the task finishes in five days.  Now, when I place this bet I have my (personal)  reasons for doing so. However, these reasons ought to relate to knowledge of the fair odds involved in the said bet.  Such fair odds can only be derived from knowledge of what would happen in a (possibly hypothetical) sequence of experiments.

The key assumption in the above argument is that my personal odds aren’t arbitrary – I should be able to justify them to another (rational) person.

Let’s look at the support interpretation. In this case I have hard evidence for stating that there’s a 75% chance of finishing in 5 days. I can take this hard evidence as my personal degree of belief (remember, as stated in the previous paragraph, any personal degree of belief should have some such rationale behind it.) However, since it is based on hard evidence, it should be rationally justifiable and hence can be associated with a sequence of experiments.

So what?

The main point from the above is the following: probabilities may be interpreted in different ways, but they have an underlying unity. That is, when we state that there is a 75% probability of finishing a task in 5 days, we are implying all the following statements (with no preference for any particular one):

1. If we were to do the task several times over, it will finish within five days in three-fourths of the cases. Of course, this will hold only if the task is done a sufficiently large number of times (which may not be practical in most cases)
2. We are willing to place a bet given 3:1 odds of completion within five days.
3. We have some hard evidence to back up statement (1) and our betting belief (2).

In reality, however,  we tend to latch on to one particular interpretation depending on the situation. One is unlikely to think in terms of hard evidence when one is buying a lottery ticket but hard evidence is a must when estimating a project. When tossing a coin one might instinctively use the frequency interpretation but when estimating a task that hasn’t been done before one might use personal belief. Nevertheless, it is worth remembering that regardless of the interpretation we choose, all three are implied. So the  next time someone gives you a probabilistic estimate, ask them if they have the evidence to back it up for sure,  but don’t forget to  ask  if they’d be willing to accept a bet based on their own stated odds.

Filed under: Bias, Estimation, General Management, Management, Probability, Project Management, Risk analysis, Statistics

Introduction

Many high profile projects fail because they succeed. This paradoxical statement is true because many projects are ill-conceived efforts directed at achieving goals that have little value or relevance to their host organisations.  Project management focuses on ensuring that the project goals are achieved in an efficient manner. The goals themselves are often “handed down from above”, so the relevance or appropriateness of these is “out of scope” for the discipline of project management.  Yet, the prevalence of projects of dubious value suggests that more attention needs to be paid to “front-end” decision making in projects – that is, decision making in the early stages, in which the initiative is just an idea.  A paper by Terry Williams and Knut Samset entitled, Issues in front-end decision making on Projects looks at the problems associated with formulating the “right” project. This post is a summary and review of the paper.

What is the front-end phase of the project?  According to Williams and Samset, “The front-end phase commences when the initial idea is conceived and proceeds to generate information, consolidate stakeholders’ views and positions, and arrive at the final decision as to whether or not to finance the project.”

Decisions made in the early stages of a project are usually more consequential than those made later on. Most major parameters – scope, funding, timelines etc. are more or less set in stone by the time a project is initiated. The problem is that these decisions are made at a time when the availability of relevant information is at its lowest.  This highlights the role of sound judgement and estimation in decision making.  Furthermore, these decisions may have long-term consequences for the organisation, so due care needs to be given to alignment of the project concept with the organisation’s strategic goals.   Finally, as the cliché goes, the only constant is change: organisations exist in ever-changing environments, so projects need to have the right governance structures in place to help navigate through this turbulence. The paper is an exploration of some of these issues as they relate to front-end decision making in projects.

Defining the project concept

Williams and Samset define the terms concept as a mental construction that outlines how a problem will be solved or a need satisfied.  Note that although the definition seems to imply that the term concept equates to technical approach, it is more than that. The project concept also includes considerations of the outcomes and their impact on the organisation and its environment.

The authors emphasise that organisations should to conceive several distinct concepts prior to initiating the project.  To this end, they recommend having a clearly defined “concept definition phase” where the relevant stakeholders create and debate different concepts. Choosing the right concept is critical because it determines how the project will be carried out, what the end result is and how it affects the organisation. The authors emphasise that the concept should be determined on the basis of the required outcome rather than the current (undesirable) situation.

When success leads to failure

This is the point alluded to in the introduction: a project may produce the required outcomes, but still be considered a failure because the outcomes are not aligned with the organisation’s strategy.  Such situations almost always arise because the project concept was not right. The authors describe an interesting example of such a project, which I’ll quote directly from the paper:

One such example is an onshore torpedo battery built inside the rocks on the northern coast of Norway in 2004 (Samset, 2008a). The facility was huge and complex, designed to accommodate as many as 150 military personnel for up to three months at a time. It was officially opened as planned and without cost overrun. It was closed down one week later by Parliamentary decision. Clearly, a potential enemy would not expose its ships to such an obvious risk: the concept had long since been overtaken by political, technological, and military development. What was quite remarkable was that this project, which can only be characterized as strategic failure, got little attention in the media, possibly because it was a success in tactical terms.

A brilliant example of a successful project that failed! The point, of course, is that although the strategic aspects of projects are considered to be outside the purview of project management,  they must be given due consideration when the project is conceptualized. The result of a project must be effective for the organisation, the efficiency of project execution actually matters less.

Shooting straight – aligning the project to strategic goals

Aligning projects with strategic goals is a difficult because the organizational and social ramifications of a project are seldom clear at the start. This is because the problem may be inherently complex – for example, no one can foresee the implications of an organizational restructure (no, not even those expensive consultants who claim to be able to).  Further, and perhaps more important, is the issue of social complexity:  stakeholders have diverse, often irreconcilable, views on what needs to be done, let alone how it should be done.  These two factors combine to make most organizational issues wicked problems.

Wicked problems have no straightforward solutions, so it is difficult if not impossible to ensure alignment to organizational strategy. There are several techniques that can be used to make sense of wicked problem. I’ve discussed one of these – dialogue mapping – in several prior posts. Paul Culmsee and I will elaborate on this  and other techniques to manage wicked problems in a forthcoming book entitled “Beyond Best Practices” – see this post for more on the book and its main theme.

One has to also recognize that the process of alignment is messier still because politics and self interest  play a role, particularly when the stakes are high. Further, at the individual level, decisions are never completely objective and are also subject to cognitive bias – which brings me to the next point…

Judgement and the formulation of organizational strategy

Formulating organizational strategy depends on making informed and accurate judgements about the future. Further, since strategies typically cover the mid to long term, one has to also allow some flexibility for adjustments along the way as one’s knowledge improves.

That’s all well and good, but it doesn’t take into account the fact that decision making isn’t a wholly rational process – humans who make the decisions are, at best, boundedly rational (sometimes rational, sometimes not).  Bounded rationality manifests itself through cognitive biases – errors of perception that can lead us to making incorrect judgements. See my post on the role of cognitive bias in project failure for more on how these biases have affected high profile projects.

The scope for faulty decision making (via cognitive bias or any other mechanism) is magnified when one deals with wicked problems. There are a number of reasons for this including:

1. Cause-effect relationships are often unclear.
2. No one has complete understanding of the problem (the problem itself is often unclear).
3. Social factors come into play (Is it possible make an “unbiased” decision about a proposed project that is going to affect one’s livelihood?)
4. Consequent from points 1 through 3,  stakeholders perceive the problem (and its solution) differently.

It is worth pointing out that project planning is generally “less wicked” than strategy formulation because the former involves more clear cut goals (even though they may be wrong-headed). There is more scope for wickedness in the latter because there are many more unknowns and “unknowables.”

Why estimates are incorrect

Cost is a major factor in deciding whether or not a project should go-ahead. Unfortunately, this is another front-end decision; one which needs to be made when there isn’t enough information available. In his book, Facts and Fallacies of Software Engineering, Robert Glass names poor estimation as one of the top causes of project failure.  This is not to say that things haven’t improved. For example, Agile methods which advocate incremental/iterative development continually refine initial estimates based on actual, measurable progress.

Techniques such as reference class forecasting have been proposed to improve estimation for projects where incremental approaches are not possible (infrastructural projects, for example). However, this technique is subject to the reference class problem.

Finally, all the aforementioned techniques assume that reliable information on which estimates can be based is a) available and b) used correctly.  But this is just where the problem lies: the two key factors that lead to poor estimation are a) the lack of knowledge about what exactly the work entails and b) the fact that people may misunderstand or even misrepresent the information if it is available.

Governance in an ever-changing environment

A negative consequence of the quest for organizational agility and flexibility is that organizational environments are turbulent. The main point of the paper is that traditional project management  (as laid out in frameworks such as PMBOK) Is ill-suited to such environments. As they state:

The key point in this article, however, is that the environment in which most projects operate is complex and turbulent, and conventional project management is not well suited to such conditions, despite the attraction of project organization to companies in fast-moving environments seeking agility and responsiveness…

Yet, ironically, this uncertainty is the reason for the spectacular growth in adoption of project management methodologies (see this post for a discussion of a relevant case study).

For project management to be truly useful, it must be able to cope with and adapt to turbulent environments. To this end, it may be best to view project management as a set of activities that emerge from a real need rather than an arbitrary imposition dictated by methodologies that are divorced from reality. This is nothing new: iterative/incremental methods, which advocate adaptation of methods to suit the environment, are a step in this direction.

Adaptive methods are obviously easier to apply on smaller projects than larger ones. However, one could argue that the need for flexibility and adaptability is even greater on massive megaprojects than smaller ones. A major consequence of a changing environment is that people’s views on what needs to be done diverge. Recent work in applying dialogue mapping to large project environments shows that it is possible to reduce this divergence. Getting people on the “same page” is, I believe, the first step to successful governance, particular in unstable environments.

Lack of information

The most important decisions on projects have to be made upfront, when little or no reliable information is available. The authors argue that the lack of information can actually be a benefit in front-end decision for the following reasons:

1. Too much information can lead to confusion and analysis paralysis.
2. Information can get out of date quickly – forecasts based on outdated information can be worse than useless because they can mislead.
3. It is often hard to tell between important and irrelevant information. The distinction may only become clear as the project proceeds.

Be that as it may, one cannot deny that front-end decision making is hampered by the lack of relevant information. The real problem, though, is that decisions are often made by those who cannot tell the difference between what’s important and what’s not.

Conclusion

The article is an excellent summary of the major impediments in front-end decision making on projects. Such decisions have a major impact on how the project unfolds, yet they are often made with little or no consideration of what exactly the project will do for the organisation, or what its impact will be.

In my experience, front-end decisions are invariably made in an ad-hoc manner, rooted more in hope and fantasy than reality.  A first step to ensuring that organizations do the right project is to ensure that all stakeholders have a common understanding of the goals of the project – that is, what needs to be done. The next is to ensure a common understanding of how those goals will be achieved. Such stakeholder alignment is best achieved using communication-centric, collaborative techniques such as dialogue mapping. Only then,  after ensuring that one is doing the right project,  does it make sense to focus on doing the project right.

Filed under: Bias, Consulting, Corporate IT, Estimation, IT Management, Management, Paper Review, portfolio management, Project Management

Introduction

The fundamental premise behind best practices  is that it is possible to reproduce the successes of those who excel by imitating them. At first sight this assumption seems obvious and uncontroversial. However, most people who have lived through an implementation of a best practice know that following such prescriptions does not guarantee success. Actually, anecdotal evidence suggests the contrary:  that most attempts at implementing best practices fail. This paradox remains unnoticed by managers and executives who continue to commit their organisations to implementing best practices that are, at best, of dubious value.

Why do best practices fail?   There has been a fair bit of research on the shortcomings of best practices, and the one thing it tells us is that there is no simple answer to this question.  In this post I’ll discuss this issue, drawing upon an old (but still very relevant) paper by Jonathan Wareham and Han Cerrits entitled, De-Contextualising Competence: Can Best Practice be Bundled and Sold.   Note that I will not cover the paper in its entirety;  my discussion will focus only on those aspects that relate to the question raised above.

I may as well say it here:  I have a secondary aim  (or more accurately,  a vested interest)  in discussing this paper.  Over the last few months Paul Culmsee and I have been working on a book that discusses reasons why best practices fail and proposes some practical techniques to address their shortcomings.  I’ll end this post with a brief discussion of the background and content of the book (see this post for Paul’s take on the book). But let’s look at the paper first…

Background

On the first page of the paper the authors state:

Although the concept of ‘imitating excellent performers’ may seem quite banal at first glance, the issue, as we will argue, is not altogether that simple after deeper consideration. Accordingly, the purpose of the paper is to explore many of the fundamental, often unquestioned, assumptions which underlie the philosophy and application of Business Best Practice transfer. In illuminating the central empirical and theoretical problems of this emerging discipline, we hope to refine our expectations of what the technique can yield, as well as contribute to theory and the improvement of practice.

One of the most valuable aspects of the paper is that it lists some of the implicit assumptions that are often glossed over by consultants and  others who sell  and implement best practice methodologies.  It turns out that these assumptions are not valid in most practical situations, which renders the practices themselves worthless.

The implicit assumptions

According to Wareham and Cerrits, the unstated premises behind best practices include:

1. Homogeneity of organisations: Most textbooks and courses on best practices present the practices as though they have an existence that is independent of organizational context.  Put another way: they assume that all organisations are essentially the same. Clearly, this isn’t the case – organisations are defined by their differences.
2. Universal yardstick: Best practices assume that there is a universal definition of what’s best, that what’s best for one is best for all others. This assumption is clearly false as organisations have different (dare I say, unique) environments, objectives and strategies. How can a universal definition of “best” fit all?
3. Transferability:  Another tacit assumption in the best practice business is that practices can be transplanted on to receiving organisations wholesale. Sure, in recent years it has been recognized that such transplants are successful only if a) the recipient organisation undertakes the changes necessary for the transplant to work and b) the practice itself is adapted to the recipient organisation. The point is in most successful cases, the change or adaptation is so great that it no longer resembles that original  best practice. This is an important point – to have a hope in hell of working, best practices have to be adapted extensively. It is also worth mentioning that  such adaptations will succeed only if they are  made in consultation with those who will be affected by the practices. I’ll say more about this later in this post
4. Alienability and stickiness: These are concepts that relate to the possibility of extracting relevant knowledge pertaining to a best practice from a source and transferring it without change to a recipient. Alienability refers to the possibility of extracting relevant knowledge from the source. Alienability is difficult because best practice knowledge is often tacit, and is therefore difficult to codify. Stickiness refers to the willingness of the recipient to learn this knowledge, and his or her ability to absorb it. Stickiness highlights the importance of obtaining employee buy-in before implementing best practices. Unfortunately most best practice implementations gloss over the issues of alienability and stickiness.
5. Validation: Wareham and Cerrits contend that best practices are rarely validated. More often than not,  recipient organisations simply believe that they will work, based on their consultants’ marketing spiel. See this short piece by Paul Strassman for more on the dangers of doing so.

What does  “best” mean anyway?

After listing the implicit assumptions, Wareham and Cerrits argue that the conceptual basis for defining a particular practice as being “best” is weak.  Their argument hinges on the observation that it is impossible to attribute the superior performance of a firm to specific managerial practices. Why? Well, because one cannot perform a control experiment to see what would happen if those practices weren’t used.

Related to the above is the somewhat subtle point that it is impossible to say, with certainty, whether practices, as they exist within model organisations, are consequences of well-thought out managerial action or whether they are merely adaptations to changing environments. If the latter were true, then there is no best practice, because the practices as they exist in model organisations are essentially random responses to organizational stimuli.

Wareham and Cerrits also present an economic perspective on best practice acquisition and transfer, but I’ll omit this as it isn’t of direct relevance to the question of why best practices fail.

Implications

The authors draw the following conclusions from their analysis:

1. The very definition of best practices is fraught with pitfalls.
2. Environmental factors have a significant effect on the evolution and transfer(ability) of “best” practices. Consequently, what works in one organisation may not work in another.

So, can anything be salvaged?  Wareham and Cerrits think so. They suggest an expanded view of best practices which includes things such as:

1. Using best practices as guides for learning new technologies or new ways of working.
2. Using best practices to generate creative insight into how business processes work in practice.
3. Using best practices as a guide for change – that is, following the high-level steps, but not necessarily the detailed prescriptions.

These are indeed sensible and reasonable statements. However, they  are much weaker than the usual hyperbole-laden claims that accompany best practices.

Discussion

Cerrits and Johnson focus on the practices themselves, not the problems they are used to solve. In my opinion, another key reason why best practices fail is that they are applied without a comprehensive understanding of the problem that they are intended to address.

I’ll clarify this using an example:  in a quest to improve efficiency an organisation might go through a major restructure. All too often, such organisations will not think through all the consequences of the restructuring (what are the long-term consequences of outsourcing certain functions, for instance). The important point to realize is that a comprehensive understanding of the consequences is possible only if all stakeholders – management and employees – are involved in planning the restructure.  Unfortunately, such a bottom-up approach is rarely taken because of the effort involved, and the  wrong-headed perception that chaos may ensue from management actually talking to people on the metaphorical shop floor.   So  most organizations take a top-down approach, dictating what will be done, with little or no employee involvement.

Organisations focus on how to achieve a particular end. The end itself, the reasons for wanting to achieve it and the consequences of doing so remain unexplored; it is assumed that these are obvious to all stakeholders. To put it in aphoristically: organizations focus on the “how” not the on the “what” or why.”

The heart of the matter

The key to understanding why best practices do not work is to realise that many organizational problems are wicked problems: i.e., problems that are hard to define, let alone solve’s  (see this paper for a comprehensive discussion of wicked problems).  Let’s look at organizational efficiency, for example.   What does it really mean to improve organizational efficiency?   More to the point, how can one arrive at a generally agreed way to improve organizational efficiency?  By generally agreed, I mean a measure that all stakeholders understand and agree on. Note that “efficiency “is just an example here – the same holds for most other matters of strategic importance to organizations: organisational strategy is a wicked problem.

Since wicked problems are hard to pin down (because they mean different things to different people), the first step to solving them is to ensure that all stakeholders have a common (or shared) understanding of what the problem is. The next step  is to achieve a shared commitment to solving that problem. Any technique that could help achieve a shared understanding of wicked problems and commitment to solving them  would  truly deserve to be called the one best practice to rule them all.

The genesis of a collaboration

About a year ago, in a series of landmark posts entitled The One Best Practice to Rule Them All, Paul Culmsee wrote about his search for a practical method to manage wicked problems.  In the articles he made a convincing case that dialogue mapping can help a diverse group of stakeholders achieve a shared understanding of such problems.  Paul’s writings inspired me to learn dialogue mapping and use it at work. I was impressed – here, finally, was a technique that didn’t claim to be a best practice, but had the potential to address some of the really complex problems that organisations face.

Since then, Paul and I have had several conversations about the failure of best practices in to tackling issues ranging from organizational change to project management. Paul is one of those rare practitioners with an excellent grounding in theory and practice.  I learnt a lot from him in those conversations. Among other things, he told me about his experiences in using dialogue mapping to tackle apparently intractable problems (see this case study from Paul’s company, for example).

Late last year, we thought of writing up some of the things we’d been talking about in a series of joint blog posts. Soon we realised that we had much more to say than would fit into a series of posts – we probably had  enough for a book.  We’re a few months into writing that book, and are quite pleased with the way it’s turning out.

Here’s a very brief summary of the book. The first part analyses why best practices fail.  Our analysis  touches  upon diverse areas like organizational rhetoric, cognitive bias, memetics and scientific management (topics that both Paul and I have written about on our blogs).  The second part of the book presents a series of case studies that illustrate some techniques that address complex problems that organizations face. The case studies are based on our experiences in using dialogue mapping and other techniques to tackle wicked problems relating to organizational strategy and project management.  The techniques we discuss go beyond the rhetoric of best practices – they work because they use a bottom-up approach that takes into account the context and environment in which the problems live.

Now, Paul writes way better than I do. For one, his writing is laugh-out-loud funny, mine  isn’t. Those who have read his work and mine may be wondering how our very different styles will combine.  I’m delighted to report that the book is way more conversational and entertaining than my blog posts.  However, I should also emphasise that we are trying to be as rigorous as we can by backing up our claims by references to research papers and/or case studies.

We’re learning a lot in the process of writing, and are enthused and excited about the book . Please stay tuned – we’ll post occasional updates on how it is progressing.

Update (16 June 2010):

An excerpt from the book has been published here.

Filed under: Consulting, Corporate IT, General Management, Management, Organizational Culture, Paper Review, Project Management

Introduction

Risk management is an important component of all project management frameworks and methodologies, so most project managers are well aware of the need to manage risks on their projects. However, most books and training courses offer little or no guidance about the relative importance of different categories of risks.   One useful way to look at risks is by whether they pose operational or strategic threats. The former category includes risks that impact project execution and the latter those that affect project goals. A recent paper entitled, Categorising Risks in Seven Large Projects – Which Risks do the Projects Focus On? (full text available here), looks at how strategic and operational risks are treated in typical, real-life projects. This post is a summary and review of the paper.

Operational and strategic risks

For the purpose of their study, the authors of the paper categorise risks as follows:

1. Operational risk: A risk that affects a project deliverable.
2. Short term strategic risk: A risk that impacts an expected outcome of the project. That is, the results expected directly from a deliverable. For example, an order processing system (deliverable) might be expected to reduce processing time by 50% on average (outcome).
3. Long term strategic risk: A risk that affects the strategic goal that the project is intended to address. For example, an expected  strategic outcome of a new order processing system  might be to boost sales by 25% over the next 2 years.

It is also necessary to define unambiguous criteria by which risks can be assigned to one of the above categories. The authors use the following criteria to classify risks:

1. A risk is an operational risk if it can impact a deliverable that is set out in the project definition (scope document, charter etc.) or delivery contract.
2. A risk is a short-term strategic if it can have an effect on functionality that is not clearly mentioned in the project documentation, but is required in order to achieve the project objectives.
3. A risk is considered to be a long-term strategic if it affects the long-term goals of the project and does not fall into the prior two categories.

The authors use the third category as a catch-all bucket for risks that do not fall into the first two categories.

Methodology

The authors collected data from the risk registers of seven large projects. Prior to data collection, the conducted interviews with relevant project personnel to get an understanding of the goals and context of the project. Further interviews were conducted, as needed, mainly to clarify points that came up in the analysis.

A point to note is that the projects studied were all in progress, but in different phases ranging from initiation to  closure.

Results and discussion

The authors’ findings can be summed up in a line: the overwhelming majority of risks were operational. The fraction of risks that were classified as long-term strategic was less than 0.5 % of the total (with over 1300 risks were classified in all).

Why is the number of strategic risks so low? The authors offer the following reasons:

1. Strategic risks do not occur while a project is in progress: The authors argue that this is plausible because strategic risks are (or should be) handled prior to a project being given the go-ahead.  This makes sense, so in a well-vetted project strategic risks will occur only if there are substantial changes in the hosting organisation and/or its environment.
2. Long term strategic risks are not the project’s responsibility: This is a view taken by most project management methodologies: a project exists only to achieve its stated objectives; its long-term impact is irrelevant.  Put another way, the focus is on efficiency, not (organisational) effectiveness (I’ll say more about this in a future post).  The authors recommend that project risk managers need to be aware of strategic issues, even though these are traditionally out of the purview of the project. Why? Well, because such issues  can have a  major impact on how the project is perceived by the organisation.
3. Strategic risks are mainly the asset owner’s (or sponsor’s) responsibility: According to conventional management wisdom strategic risks are the responsibility of management, not the project team. In contrast, the authors suggest that the project team is perhaps better placed to identify some strategic risks long before they come to management’s attention.  From personal experience I can vouch that this is true, but would add that it can be difficult to raise awareness of these risks in a politically acceptable way.

Conclusion

The main point that the article makes is that strategic risks, though often ignored, can have a huge effect on projects and how they are viewed by the larger organisation.  It is therefore in important that these risks are identified and escalated to sponsors and other decision makers in a timely manner.  This is a message that organisations would do well to heed, particularly those that have a “shoot the messenger” culture which discourages honest and open communication about such risks.

Filed under: Consulting, IT Management, Paper Review, portfolio management, Project Management, Risk analysis

Introduction

Most of the research and practice literature on project management tends to view projects as being isolated from their environment.  It is obvious to anyone who has worked on a project that this isn’t so. In view of this,  it is useful to look at the relationship between projects and the organisations that host them.  This post looks at this issue, drawing on a paper by Gernot Grabher entitled, Cool Projects, Boring Institutions: Temporary Collaboration in Social Context.

The emergence of projects

Grabher begins his discussion with a sketch of the how projects emerged as a distinct work form. Projects  - i.e. time bound, goal focused activities – have always been around. The modern notion of a project, however,   arose from a development philosophy that came out of the US Department of Defense in the 1950s.  He states,

…Instead of fragmenting and pre-specifying the development of military technologies along functional disciplines, these technologies were described in relation to their objectives, i.e. the military parameters of these weapons. The pacing of these concentrated efforts was crucial: parameters had to be met, goals had to be accomplished according to a grand scheme (program?) to win the armament race. Development processes that earlier were seen as separate activities were now conceptualized as an integrated entity called a program, system or project. The overwhelming scale of these projects in terms of financial and scientific resources as well as their ambitious timing created formidable problems of coordination and control. Experiments with various forms of organizational control ultimately lead to the professionalization of the role of the project manager…

From thereon the concepts of projects and project management were taken up (with much enthusiasm and optimism) by business and industry. The formalization of various project management methodologies, standards , qualifications and trade journals can be seen a culmination of this process.

Given the military-industrial origins of the profession, it is easy to see why a “command and control” philosophy dominates much of project management thought and practice. Many of the early projects that are paraded as textbook examples of successful projects operated outside normal organizational oversight. They were, to a large extent, deliberately shielded from external influences.  I believe this is why isolation from the environment is seen as a Good Thing by project managers -  problems of coordination and control become so  much simpler when one does not have to manage relationships and politics that are (perceived as being) external to a project. This practice may be necessary and workable for classified projects that run on billion dollar budgets, but it doesn’t work so well in   environments that most project managers work in. Projects don’t take place in a   vacuum; they are born, live and die in real-world organizations. To forget that is to see the “tree of the project” and miss the “forest of the organization.” This is particularly so because,  unlike those near-mythical mega-projects of the 1950s,  the efforts that you and I work on are deeply entwined with their hosting organizations.

Organisation-related characteristics of projects

Grabher then notes some characteristics of projects. I summarize these in the next few paragraphs.

First, it is interesting that the original meaning of the word “project” referred to a “proposal” or “idea”, rather than a “directed, time-bound effort.” Grabher points out that this  shift in meaning was  accompanied by a shift in focus: from project as idea (or goal) to project as process (or means to achieving a goal).   Projects are thus seen as vehicles for achieving organisational goals.

Second, Grabher notes that projects are often hard to decompose into constituent tasks, and that such a (commonly agreed) decomposition is only possible when stakeholders interrelate with each other continually.  This underscores the importance of communication in projects.

Third, Grabher highlights the importance of the project manager (he uses the term contractor) as the “lynchpin on whom trust is focused.” The role of the manager is particularly important in projects on which team members do not have the time to get to know each other well.

Fourth, the project manager / contractor is also the wielder of organizational authority as far as the project is concerned. He or she is, in this sense, a representative of the organization – a person whose presence underlines the fact that the project exists to achieve specified organizational goals.

Finally, deadlines are a defining aspect of projects. They serve several functions. For example, they ensure that a sense of urgency for action and progress remains through the duration of the project. They also might serve to legitimize execution of project work without external interference (this argument was frequently used in the military-industrial projects of the 1950s). But above all, the final deadline,  which culminates in the termination of the project, also serves as a connector to the rest of the organization. It is a time in which handoffs, documentation, team disbanding etc. occurs, thus enabling the results and  experiences from the project disperse into the wider organization.

Projects in organisations

The characteristics noted above highlight the dual nature of projects: on the one hand, as noted earlier, projects are seen as semi-autonomous temporary organisations, but on the other they are also firmly embedded within the hosting organisation. An effect of the latter is particularly evident  in consulting and software services firms (or even corporate IT shops), which tend to do similar projects over and over.  As Grabher notes,

[projects] apparently operate in a milieu of recurrent collaboration that, after several project cycles, fills a pool of resources and gels into latent networks. Project organising is mostly directed towards the actual realization of a potential that is generated and reproduced by the practice of drawing on core members of (successful) prior projects to serve on derivative successor projects. Such chains of repeated co-operation are held together (or cut off ) by the reputation members gain (or lose) in previous collaborations…

Another aspect of embedded-ness is the co-location of team members within a larger organizational milieu. The standard benefits of co-location are well known. These are:

1. Savings of transactional costs such as those incurred in communication, supervision of staff at remote locations etc. See my post on a transaction cost view of outsourcing for more on this.
2. Co-location improves the efficacy of communication by encouraging face-to-face interactions.
3. It enables “near real-time” monitoring of the health of the project and its environment.

There’s more though. Grabher notes that  in addition to the above “intentional” or “strategic” benefits, co-location also ensures that  team members are exposed to the same organizational noise – which consists of  a “concoction of rumours, impressions, recommendations, trade folklore and strategic misinformation (falsehoods!).”  Co-location enables project teams to make collective sense of organisational noise – this shared understanding of the environment can contribute significantly to the creation of a team spirit.

A related notionis that of enculturation: that is, the process of becoming an accepted member of the group, or an insider. This has less to do with expertise and knowledge than learning the unspoken rules and norms of a community.  Although becoming a member of  a community has much to do with social interactions within the workplace, there is more: a lot of essential know-how and know-what is transferred through informal interactions between senior members of the team (who are often senior members of the organisation) and others.

Projects generally need to draw upon a range of organizational resources: people and physical infrastructure being the most obvious ones.   Grabher notes that the increasing projectisation of organizations can be attributed to a perception that project-based management is an efficient way to allocate productive resources in a flexible manner   (…whether this perception is correct, is another matter altogether). However, there are other less obvious influences that organisations exert too.  For example,  Grabher points out that organizational norms and rules provide the basis for the emergence of swift trust, which is trust based on roles and professional ability rather than individuals and personalities.  Further, at a higher level,  organizational culture plays a role in determining how a project is governed, managed and run. These explicit and implicit norms have a stabilising influence on projects.

In addition to the stabilizing influence of the hosting organisation, projects also offer opportunities to build and enhance links between organisations – for instance, strategic partnerships.  This is, in effect, institution building aimed at leveraging the strengths of the participating organisations for a greater joint benefit. In such situations the participating organisations take on the role of “lynchpins” on whom trust is focused.

Grabher makes the point that firms (and institutions comprised of firms) not only provide resources that make projects possible, but also host a range of processes that are needed to organize and run projects. For one, projects are usually preceded by several organisational processes involving deliberation, selection and preparation. These activities have to occur for a project to happen, but they normally fall outside the purview of the project.

A somewhat paradoxical aspect of projects is although they offer the opportunity for enhancing organizational knowledge, this rarely happens in practice. The high pressure environment in projects leaves little time for formal training or informal learning, or even to capture knowledge in documents. To  a large extent the hosting organisations are to blame: Grabher suggests that this paradox is a consequence of the lack of organizational redundancy in project-based organizing.

I’ll end this section with the observation that the social dimension of projects is often neglected.  Projects are often hindered by organizational politics and inertia.  Further, a large number of  projects fail because of varying perceptions of project goals and the rationale behind them. Although it seems obvious that a project should not proceed unless all stakeholders have a shared understanding of objectives and the reasons for them, it is surprising how many projects drift along without it.   Many project planners neglect this issue, and it invariably comes back to bite them.

Conclusions

In the conclusion to the paper, Grabher states:

The formation and operation of projects essentially relies on a societal infrastructure which is built on and around networks, localities, institutions and firms. Relations between temporary and permanent systems are not a matter of straightforward substitution but have to be regarded in terms of interdependence. ‘Cool’ projects, indeed, rely on ‘boring’ institutions…

This is unarguable, but it should also be kept in mind that projects are often subject to negative organisational influences which can slow them down, or even kill them altogether (which is perhaps why those early defence projects were set up as near-autonomous initiatives). So  although  it is true that projects  are made possible and sustained by the organisations they’re embedded in,  they are sometimes  hindered by those very organisations .

To sum up in a line:    Projects depend on organisations not only for material and human resources, but also draw sustenance from (and are affected by)  the  social environment and culture that exists within those organisations.

Filed under: Consulting, Corporate IT, IT Management, Organizational Culture, Paper Review, portfolio management, Project Management

Introduction

Managers make decisions based on incomplete information, so it is no surprise that the tools of probability and statistics have made their way into management practice. This trend has accelerated somewhat over the last few years, particularly with the availability of software tools that simplify much of the grunt-work of using probabilistic techniques such as Monte Carlo methods or Bayesian networks. Purveyors of tools and methodologies and assume probabilities (or more correctly, probability distributions) to be known, or exhort users to determine probabilities using relevant historical data. The word relevant is important: it emphasises that the data used to calculate probabilities (or distributions) should be from situations that are similar to the one at hand. This innocuous statement papers over a fundamental problem in the foundations of probability: the reference class problem. This post is a brief introduction to the reference class problem and its implications for project management.

I’ll begin with some background and then, after defining the problem, I’ll present a couple of illustrations of the problem drawn from project management.

Background and the Problem

The most commonly held interpretation of probability is that it is a measure of the frequency with which an event of interest occurs. In this frequentist view, as it is called, probability is defined as the ratio of the number of times the event of interest occurs to the total number of events. An example might help clarify what this means: the probability that a specific project will finish on time is given by the ratio of the number of similar projects that have finished on time to the total number of similar projects undertaken (including both on-time and not-on-time projects).

At first sight the frequentist approach seems a reasonable one. However, in this straightforward definition of probability lurks a problem: how do we determine which events are similar to the one at hand? In terms of the example: what are the criteria by which we can determine the projects that resemble the one we’re interested in? Do we look at projects with similar scope, or do we use size (in terms of budget, resources or other measure), or technology or….? There could be a range of criteria that one could use, but one never knows with certainty which one(s) is (are) the right one(s). Why is it an issue? It’s an issue because probability changes depending on the classification criteria used. This is the reference class problem.

In a paper entitled The Reference Class Problem is Your Problem Too, the philosopher Alan Hajek sums it up as follows:

The reference class problem arises when we want to assign a probability to a proposition (or sentence, or event) X, which may be classified in various ways, yet its probability can change depending on how it is classified.

Incidentally, in another paper entitled Conditional Probability is the Very Guide of Life, Hajek discusses how the reference class problem afflicts all major interpretations of probability, not just the frequentist approach. We’ll stick with the latter interpretation since it is the one used in project management practice and research… and virtually all the social and natural sciences to boot.

The reference class problem in project management

Let’s look at a couple of project management-related illustrations of the reference class problem.
First up, consider the technique of reference class forecasting which I’ve discussed in this post. Note that reference class forecasting technique is distinct from the reference class problem although, as we shall see in less than a minute, the technique is fatally afflicted by the problem.

What’s reference class forecasting? To quote from the post referenced earlier, the technique involves:

…creating a probability distribution of estimates based on data for completed projects that are similar to the one of interest, and then comparing the said project with the distribution in order to get a most likely outcome. Basically, [it] consists of the following steps:

1. Collecting data for a number of similar past projects – these projects form the reference class. The reference class must encompass a sufficient number of projects to produce a meaningful statistical distribution, but individual projects must be similar to the project of interest.
2. Establishing a probability distribution based on (reliable!) data for the reference class. The challenge here is to get good data for a sufficient number of reference class projects.
3. Predicting most likely outcomes for the project of interest based on comparisons with the reference class distribution.

Now, the key assumption in reference class forecasting is that it is possible to identify a number of completed projects that are similar to the one at hand. But what does “similar” mean? Clearly the composition of the reference class depends on the similarity criteria used, and consequently so does the resulting distribution. Reference class forecasting is a victim of the reference class problem!

The reference class problem will affect any technique that uses arbitrary criteria to determine the set of all possible events. As another example, the probability distributions used in Monte Carlo simulations (of project cost, duration or whatever) are determined using historical data. Again, typically one selects projects (or tasks – if one is doing a task level simulation) that are similar to the one at hand. Defining “similar” is left to common sense or expert judgement or some other subjective approach. Yet, by the most commonly used definition, a project is a “temporary endeavor, having a defined beginning and end, undertaken to meet unique goals and objectives”. By definition, therefore, we never do the same project twice – at best we do the same project differently  (and the same applies to tasks). So, despite ones best intentions and efforts, historical data can never be totally congruent to the situation at hand. There will always be differences, and one cannot tell with certainty that those differences do not matter.

Truth be told, most organizations do not retain data on completed projects – except superficial stuff that isn’t much use. The reference class problem seems to justify the position of this slack majority. After all, why bother keeping data when one isn’t able to use it to predict project performance. This argument is wrong-headed: although one cannot use it to calculate probabilities, historical data is useful because it keeps us from repeating our errors.  Just don’t expect the data to yield reliable quantitative information on probabilities.

Before I close this piece, I should clarify that there are areas in which the reference class problem is not an issue. In physics, for example, the discipline of statistical mechanics is founded on the principle that the average motion of large collections of molecules can be treated statistically. Clearly, there is no problem here: molecules are indeed indistinguishable from each other, so it is clear that a particular molecule (of a gas in a container of carbon dioxide, say) belongs to the reference class of all carbon dioxide molecules in that container. In general this is true of any situation where one is dealing with a large collection of identical (or very similar) entities.

Conclusion

The reference class problem affects most probabilistic methods in project management  and  other areas of the social sciences. It is a problem because it is often impossible to know beforehand  which attributes of the objects or events of interest are the most significant ones. Consequently it is impossible to determine with certainty whether or not a particular object or event belongs to a defined reference class.

I’ll end with an anecdote to illustrate my point:

Some time ago I was asked to provide estimates for design work that was superficially similar to something I’d done before. “You’ve done this before,” a manager said, “so you should be able to estimate this quite accurately.”

As many best practices and methodologies recommend, I used a mix of historical data and “expert” judgement (and added in a dash of padding) to arrive at (what I thought was) a robust estimate. To all you agilists out there, an incremental approach was not an option in this case.

I got it wrong – badly wrong. It turned out that the unique features of the project, which weren’t apparent at first, made a mockery of my estimates. I didn’t know it then, but I’d fallen victim to the reference class problem.

Finally, it should be clear that although my examples are project management focused, the arguments are quite general. They apply to all areas of management theory and practice, and indeed to most areas of inquiry that use probabilistic techniques. To use the words of Alan Hajek:  the reference class problem is your  problem too.

Filed under: Estimation, Probability, Project Management, Statistics Tagged: Monte Carlo Simulation

Introduction

I’ll begin with an example. Assume you’re having a dishwasher installed in your kitchen. This (simple?) task requires the services of a plumber and an electrician, and both of them need to be present to complete the job. You’ve asked them to come in at 7:30 am. Going from previous experience, these guys are punctual 50% of the time. What’s the probability that work will begin at 7:30 am?

At first sight, it seems there’s a 50% chance of starting on time. However, this is incorrect – the chance of starting on time is actually 25%, the product of the individual probabilities for each of the tradesmen. This simple example illustrates the central theme of a book by Sam Savage entitled, The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty. This post is a detailed review of the book.

The key message that Savage conveys  is that uncertain quantities cannot be represented by single numbers, rather they are  a range of numbers each with a different probability of occurrence. Hence such quantities cannot be manipulated using standard arithmetic operations. The example mentioned in the previous paragraphs illustrate this point. This is well known to  those who work with uncertain numbers (actuaries, for instance), but is not so well understood by business managers and decision makers. Hence the executive who asks his long-suffering subordinate to give him a projected sales figure for next month, with the quoted number then being taken as the 100% certain figure.  Sadly such stories are more the norm than the exception,  so it is clear that there is a need for a better understanding of how uncertain quantities should be interpreted. The main aim of the book is to help those with little or no statistical training achieve that understanding.

Developing an intuition for uncertainty

Early in the book, Savage presents five tools that can be used to develop a feel for uncertainty. He refers to these tools as mindles – or mind handles.  His five mindles for uncertainty are:

1. Risk is in the eye of the beholder, uncertainty isn’t. Basically this implies that uncertainty does not equate to risk. An uncertain event is a risk only if there is a potential loss or gain involved. See my review of Douglas Hubbard’s book on the failure of risk management for more on risk vs. uncertainty.
2. An uncertain quantity is a shape (or a distribution of numbers) rather than a single number. The broadness of the shape is a measure of the degree of uncertainty. See my post on the inherent uncertainty of project task estimates for an intuitive discussion of how a task estimate is a shape rather than a number.
3. A combination of several uncertain numbers is also a shape, but the combined shape is very different from those of the individual uncertainties.  Specifically, if the uncertain quantities are independent, the combined  shape can be narrower (i.e. less uncertain) than that of the individual shapes.  This provides the justification for portfolio diversification, which tells us not to put all our money on one horse, or eggs in one basket etc. See my introductory post on Monte Carlo simulations to see an example of how multiple uncertain quantities can combine in different ways.
4. If the individual uncertain quantities (discussed in the previous point) aren’t independent, the overall uncertainty can increase or decrease depending on whether the quantities are positively or negatively related. The nature of the relationship (positive or negative) can be determined from a scatter plot of the quantities. See my post on simulation of correlated project tasks for examples of scatter plots. The post also discusses how positive relationships (or correlations) can increase uncertainty.
5. Plans based on average numbers are incorrect on average. Using average numbers in plans usually entails manipulating them algebraically and/or plugging them into functions. Savage explains how the form of the function can lead to an overestimation or underestimation of the planned value. Although this sounds a somewhat abstruse, the basic idea is simple: manipulating an average number using mathematical operations will amplify the error caused by the flaw of averages.

Savage explains the above concepts using simple arithmetic supplemented with examples drawn from a range of real-life business problems.

The two forms of the flaw of averages

The book makes a distinction between two forms of the flaw of averages. In its  first avatar, the flaw states that  the combined average of two uncertain quantities equals the sum of their individual averages, but the shape of the combined uncertainty can be very different from the sum of the individual shapes (Recall that an uncertain number is a shape, but its average is a number).  Savage calls this the weak form of the flaw of averages. The weak form applies when one deals with uncertain quantities directly.  An example of this is when one adds up probabilistic estimates for two independent project tasks with no lead or lag between them. In this case the average completion time is the sum of the average completion times for the individual tasks, but the shape of the distribution of the combined tasks does not resemble the shape of the individual distributions. The fact that the shape is different is a consequence of the fact that probabilities cannot be “added up” like simple numbers. See the first example in my post on Monte Carlo simulation of project tasks for an illustration of this point.

In contrast, when one deals with functions of uncertain quantities, the combined average of the functions does not equal the sum of the individual averages. This happens because functions “weight” random variables in a non-uniform manner, thereby amplifying certain values of the variable. An example of this is where we have two sequential tasks with an earliest possible start time for the second. The earliest possible start time for the second task introduces a nonlinearity in cases where the first task finishes early (essentially because there is a lag between the finish of the first task and the start of the second in this situation). The constraint causes the average of the combined tasks to be greater than the sum of the individual averages. Savage calls this the strong form of the flaw of averages. It applies whenever one deals with nonlinear functions of uncertain variables. See the second example in my post on Monte Carlo simulation of multiple project tasks for an illustration of this point.

Much of the book presents real-life illustrations of the two forms of the flaw in risk assessment, drawn from finance to the film industry and  from petroleum to pharmaceutical supply chains. He also covers the average-based abuse of statistics in discussions on topical “hot-button” issues such as climate change and health care.

De-jargonising statistics

A layperson-friendly feature of the book is that it explains statistical terms in plain English. As an example, Savage spends an entire chapter demystifying the term correlation using scatter plots . Another term that he explains is the Central Limit Theorem (CLT), which states that the sum of independent random variables resembles the Normal (or bell-shaped) distribution.  A consequence of CLT is that one can reduce investment risk by diversifying one’s investments – i.e. making several (small) independent investments rather than a single (large) one  – this is essentially mindle # 3 discussed earlier.

Decisions, decisions

Towards the middle of the book, Savage makes a foray into decision theory, focusing on the concept of value of information. Since decisions are (or should be) made on the basis of information, one needs to gather pertinent information prior to making a decision. Now, information gathering costs money (and time, which translates to money). This brings up the question as to how much should one spend in collecting information relevant to a particular decision? It turns out that in many cases one can use decision theory to put a dollar value on a particular piece of information.  Surprisingly it turns out that organisations often  over-spend in gathering irrelevant information. Savage spends a few chapters discussing how one can compute the value of information based on simple techniques of decision theory. As interesting as this section is, however, I think it is a somewhat disconnected from the rest of the book.

Curing the flaw: SIPs, SLURPS and Probability Management

The last part of the book is dedicated to outlining a solution (or as Savage calls it, a cure) to average-based – or flawed – statistical  thinking. The central idea is to use pre-generated libraries of simulation trials for variables of interest. Savage calls such a packaged set of simulation trials a Stochastic Information Packet (SIP). Here’s an example of how it might work in practice:

Most business organisations worry about next year’s sales. Different divisions in the organisation might forecast sales using different of techniques. Further, they may use these forecasts as the basis for other calculations (such as profit and expenses for example). The forecasted numbers cannot be compared with each other because each calculation is based on different simulations or worse, different probability distributions.  The upshot of this is that forecasted sales results can’t be combined or even compared. The problem can be avoided if everyone in the organisation uses the same SIP  for forecasted sales. The results of calculations can be compared, and even combined, because they are based on the same simulation.

Calculations that are based on the same SIP (or set of SIPs) form a set of simulations that can be combined and manipulated using arithmetic operations. Savage calls such sets of simulations, Scenario Library Units with Relationships Preserved (or SLURPS).  The name reflects the fact that each of the calculations is based on the same set of sales scenarios (or results of simulation trials).  Regarding the terminology: I’m not a fan of laboured acronyms, but concede that they can serve as a good mnemonics.

The proposed approach ensures that the results of the combined calculations will avoid the flaw of averages,and exhibit the correct statistical behaviour. However, it assumes that there is an organisation-wide authority responsible for generating and maintaining appropriate SIPs.  This authority – the probability manager -  will be responsible for a “database” of SIPs that covers all uncertain quantities of interest to the business, and make these available to everyone in the organisation who needs to use them.   To quote from the book, probability management involves:

…a data management system in which the entities being managed are not numbers, but uncertainties, that is, probability distributions. The central database is a Scenario Library containing thousands of potential future values of uncertain business parameters. The library exchanges information with desktop distribution processors that do for probability distributions what word processors did for words and what spreadsheets did for numbers.

Savage  sees probability management as a key step towards managing uncertainty and risk in a coherent manner across organisations.  He  mentions that  some organizations that have already started down this route (Shell and Merck, for instance). The book can thus also be seen as a manifesto for the new discipline of probability management.

Conclusion

I have come across the flaw of averages in various walks of organizational life ranging from project scheduling to operational risk analysis. Most often, the folks responsible for analysing uncertainty are aware of the flaw, and have the requisite knowledge of statistics to deal with it. However, such analyses can be hard to explain to those who lack this knowledge.  Hence managers who demand a single number. Yes, such attitudes betray a lack of understanding of what uncertain numbers are and how they can be combined, but that’s the way it is in most organizations. The book is directed largely to that audience.

To sum up:  the book is an entertaining and informative read on some common misunderstandings of statistics. Along the way  the author translates many statistical principles and terms from “jargonese” to plain English. The book deserves to  be read widely, especially by those who need it the most: managers and other decision-makers who need to understand the arithmetic of uncertainty.

Filed under: Estimation, Probability, Project Management, Risk analysis, Statistics Tagged: Book review

Introduction

When can one say that a project has  failed?  Although the answer to this question depends on the criteria used to measure project success, most project managers would not think the question (or its answer) could raise much controversy.  In this post I discuss why the issue of project success is more ambiguous (and contentious!) than seems at first sight.

The crux of the issue lies in the observation that managers and programmers often use different criteria to judge project outcomes. As Robert Glass states in his wonderful book on facts and fallacies of  software engineering:

There is a disconnect between management and their programmers. In one research study of a project that failed to meet its estimates and was seen by its management as a failure, the technical participants saw it as the most successful project they had ever worked on.

I’ll expand on this statement, drawing from Glass’ book, the research study he mentions, and personal experience.

The disconnect

Much of  Glass’ discussion is based on a paper by Kurt Linberg entitled, Software Developer Perceptions about Software Project Failures: A Case Study.  The  paper details a lot of information about the project, its context, reporting structures and the composition of the team.  Although this is important from the perspective of a research study, it is more relevant to look at how the various project stakeholders perceived the project.

First a few words about the project that Linberg studied. The project was aimed at creating a software-driven medical device. The management-approved schedule for the project ran over 14 months, but the project actually took 27 months to complete.  Further, the software development portion of the product was originally budgeted at a little over a million dollar but ended up costing more than four million! In view of this there’s little surprise that management deemed the project a failure.

What about the programmers? Linberg interviewed/surveyed many of the developers who were a part of the team. One of the questions he asked was: what was the most successful project that you have worked on, and why? Surprisingly, more than 50% of the team answered that the project described above was the most successful one they had ever worked on! The reasons they gave were revealing:

1. The project was a technical challenge.
2. The product worked as intended.
3. The team was small and high performing.

The following quote from one of the team members says it all:

[the project] was the most successful because of what we accomplished. The code was more solid than any I have ever written. The verification testing was extensive. All the other places that I’ve worked relied on the software developer doing the testing. I always thought that I could have done more testing. I also really enjoyed the people I worked with. I also think that it was the best-managed project of any that I’ve experienced. Also, the product was well designed. I did not see the scope creep that is so prevalent on other projects. I never felt pressure from schedule. I would ask the project manager and technical lead if we were in trouble. They would be cool, calm and collected. They emphasized the importance of doing the best job we could do. We were able to focus our energy on the tasks at hand!

So, we see a complete disconnect: according to (majority of) the developers the project was success because it delivered a quality product, but by management metrics of budget and time it was a failure!

Bridging the gap

To bridge the disconnect between management and developers, Linberg asked the team members why the project was late and over-budget. The following reasons were offered:

1. Unrealistic schedule and expectations.
2. Poor understanding of scope (underestimating the effort required)
3. Lack of resources

One of the team members summarised the schedule situation as follows:

Unfortunately, I believe program management established over-ambitious and incorrect expectations during early meetings with the executives. In reality, we had no idea how long it would take to develop this system. It is a real tribute to this team that they pulled together and developed an excellent product.

Another had this to say about the poor understanding of scope / lack of resources:

We ignored history. Although we didn’t have experience estimating this type of development, there are groups of people in our building that have been working these types of applications for years. They typically have 6 to 10 people working on their applications. How could management not see that assigning one or two people and constraining the schedule would be unrealistic?

Yet, the team thought that the project was managed quite well. Quoting from the paper:

Compared with other projects, the software developers believed the project management on the project was between average and best. In two cases, the software developers said it was the best that they had experienced.  One of the software developers that gave high marks for the project management stated that the project manager had good technical skills and could manage time lines.

In fact, one of the developers said:

The project manager, team lead, and program manager each provided leadership. The leadership worked very well together. The program manager knew his strengths and his boundaries. He left the software and firmware management to the software project manager. The software project manager was not threatened and left technical decisions to the team lead. We were always told of changes before they happened. The leadership was the best that I have experienced in my 14 years.

The overall impression that Linberg conveys is that the project was well-managed.   So where is the problem?

The programmers felt that upper management was largely responsible for the disconnect. This makes sense: front line managers rarely have the clout to set overall allocations of time and resources. They have to work within the parameters set by executives.

Based on this, Linberg proposes a new definition of software project success, which is based on industry averages rather than arbitrary metrics.

Completed Cancelled Failure Does not meet customer expectations Not learning anything that can be applied to future projects Low Success Below average cost, effort, and schedule performance compared to industry AND meeting quality expectations Some learning can be applied to future projects Success Average cost, effort, and schedule performance compared to industry AND meeting quality expectations Some learning can be applied to future projects and some artifacts can be used on future projects. High Success Better than average cost, effort, and schedule performance compared to industry AND meeting quality expectations Substantial learning can be applied to future projects and a large number of artifacts used. Exceptional Success Meeting all quality, cost, effort and schedule expectations. A Cancelled project cannot be considered an exceptional success.

This definition begins to bridge the chasm between developer and senior management perceptions of project success. It does so by broadening the definition of success. The current project management-driven definition is too narrow because it takes only one perspective into account – the perspective of those who hold the purse-strings.  Such a definition cannot work because it completely ignores the viewpoint of those who know software development best: the developers.

Conclusion

Linberg’s case study highlights the fact that there can be a huge gap between how developers and managers view project outcomes. Interestingly, the factors that cause the gap are usually present from the start of the project: unrealistic schedules, poorly understood scope and lack of adequate resources. One could hypothesise that most of the projects that “fail” do so for one or more of these reasons. Also interesting is the fact that even such “failed” projects can be deemed successful in the eyes of developers because they see things from a different perspective: what was learnt and what can be used again. Seen in this light, a project that goes over-time and over-budget can be successful because the experience gained can be used on future projects.

I’ll end with a story from experience. Some years ago I was a part of a team developing a product for a telecom software company. The product was ambitious in scope, and the timeline and budget tight. Nevertheless,  the team looked forward to the challenge of designing and building a product from scratch. The architect, project manager  and several members of the development team (excluding me) had years of experience in building software for the telecom industry. The product specs were developed by domain experts.  Management’s initial intentions were good: they assured the dev team that they (management) knew that it would take time to develop a solid product from the ground up. To reduce risk it was decided to build the product using incremental approach with monthly deliverables.

Progress was slow but steady.  Unfortunately, as the product started taking shape, business reality intruded: the sales guys who had been working hard  had not been able to convince anyone to sign up.  Funds began to dry up and management responded by cracking the whip.  Predictably, the project started to unravel and was canned a few months later.  Officially the project was deemed a failure, yet many  on the dev team believed that  a) it was one of the best projects they had worked on b) they had learnt a lot and  c) what they had learnt wouldn’t go waste.

Was the project a failure? I suppose the answer is that it failed from a purely  commercial standpoint. From a broader perspective, however, it fits quite neatly into Linberg’s definition of a successful cancelled project.

Perhaps it is time to rethink the definition of project success.

Filed under: Corporate IT, IT Management, Organizational Culture, Paper Review, Project Management