In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), my tenth and final reason for using Agile is that Agile addresses the realities of software development and the needs of the business. This post will expand upon this topic.

One reality of software development is that there is still too much unsustainable development in the industry as a whole. Squeezed projects with significant overtime and stress prevent us from getting smarter, let alone working smarter. Technology and business are continuing to change at ever-increasing rates; keeping pace with the business while keeping our skills current and relevant is a major challenge, yet essential if we are going to improve productivity. Through sustainable development and continuous improvement, Agile development helps teams and individuals do both.

A few tangible benefits associated with Agile development that meet the realities of both software development and the business – contrasted against traditional, plan-driven development projects – can be categorized broadly as: The early bird gets the worm.

Software project teams need feedback about features from the users. No matter what documentation was prepared in advance of development, there is nothing like having the users seeing the working software to obtain final, definitive agreement on whether the software meets their needs. Signing off on a specification and “building to the spec” might meet a project measurement, but it won’t necessarily satisfy the customer.

Why? In my experience, I’ve found that users aren’t particularly good at dealing in the abstract, or at least they aren’t as good at dealing in the abstract as software developers. They need to see things, not conceptualize them. That, coupled with the fact that communication between individuals is always subject to misinterpretation at some level, leads to the ever-present “requirements problems.”

I’ve heard the phrase, “Now that I see it…” quite often over the years, followed by a change request. Agile development’s frequent delivery of features enables users to routinely inspect the (working) software to provide immediate feedback. And because the highest-priority features are delivered first, software project teams get early feedback on the most valuable business features, allowing for course-corrections to occur early, instead of late, in the project.

Because the highest-priority features are being worked to completion early, the business can benefit from an early return on investment. With traditional, plan-driven projects, software is delivered as a complete package, but the business must wait until the end of the project to begin realizing a benefit.

Another, related benefit to using Agile development is that valuable time is not spent on the details of features that may or may not be a priority later. With traditional, plan-driven projects, you run the risk of investing time and energy into defining and designing features that may not be desirable or even needed. Time is money, and I prefer not to waste either.

Early delivery, early feedback, and an early return on investment; the early bird (the business) does get the “worm” with Agile development – or is it really a caterpillar that morphs into a butterfly in the cocoon of the Agile team?

Another reality is that business is rapidly changing. As Faisal Hoque points out in an article, The Speed of Business Today, “Constant change is the new dynamic of the global economy, and makes agility even more necessary than at any point in business history.”

Software development should align itself to the needs of the business by being able to respond change, and Agile development does just that. In the 2009 State of Agile Development Survey sponsored by VersionOne, the number one benefit obtained from implementing Agile development was the ability to manage changing priorities. 90 percent of the respondents said implementing Agile either improved or significantly improved their ability to manage changing priorities. (The survey data included 2,570 participants in 88 countries.) An older survey, the Shine Technologies 2003 Survey, was another global survey of actual experiences using Agile development. The highest ranked positive feature reported in the survey was the ability to “Respond to change over plan.”

In terms of general alignment between the business and IT, the 3rd-highest benefit obtained from using Agile development reported in the 2009 State of Agile Development Survey was an Improved Alignment Between IT and Business Objectives. This is supported by the Shine Technologies 2003 Survey, which reported that 83 percent of those surveyed stated that business satisfaction was better or significantly better.

Satisfaction can be a function of a variety of factors, but one reality is that software is being designed and built for the customer. People like to feel involved, that their needs are paramount in your mind. Traditional, plan-driven projects “respond” to the customer, but usually with change control and a “here’s how the project will be impacted” assessment. A rigid, process-oriented approach isn’t bad, but it can develop into an almost adversarial relationship where the customer feels that they are the one that must press for changes to their software.

This is because traditional, plan-driven projects have a focus on “planning the work and working the plan” where delivering to the specification is the key measurement. While there are project managers who run their projects as a series of negotiations (which helps maintain the relationship between the team and the customer), the very nature of the process causes people to lock in on what is defined – and change is becomes disruptive.

Agile development is designed to handle change; people don’t become wedded to a plan because they haven’t invested a lot of time and effort defining, estimating, and designing everything prior to actual construction. Agile teams wait until the last responsible moment to begin diving into the details of any one feature. And features that haven’t been started yet are considered negotiable at all times. This allows the business to change its mind with ease. And because Agile teams work with the customer throughout the project, the customer feels involved and valued throughout the entire process.

Agile development also stresses the need for the use of sound technical practices as a means of improving productivity. While you don’t have to be on an Agile team to make use of these practices, Agile development promotes them. Automated testing, Test-Driven Development, continuous integration are all geared towards improving throughput without sacrificing quality or sleep.

My final point, and final reality, is that software professionals are all adults, and they should be treated as such. Agile development, with its autonomous, self-directed teams allows people to define and manage their own work, which provides a motivational – and productive – boost.

Dan Pink, in his book Drive: The Surprising Truth About What Motivates Us, cited a study conducted by researchers at Cornell University that examined 320 small businesses, half of which granted the workers autonomy, the other half relying on top-down direction. The businesses that offered autonomy grew at four times the rate of the control-oriented firms and had one-third the turnover.

Top-down, control-oriented management brings its share of problems. In his book, The Way We’re Working Isn’t Working, Tony Schwartz puts this into solid perspective. Schwartz states, “The all-too common dynamic is today’s workplace is parent-child. Most employers tell employees when to come to work, when to leave, and how they’re expected to work when they’re at the office. Treated like children, many employees unconsciously adopt the role to which they’ve been consigned. Feeling disempowered and vulnerable, they lose the will and confidence to take real initiative or to think independently. Doing what they’re expected to do often becomes more important than doing what makes most sense.”

Command-and-control management creates the need to manage people more closely! Is that what we want? I know that I don't. Agile development's final contribution to the realities of software development is that it provides the opportunity for everyone to be treated as valued, professional, adults. That works for me!

In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), my ninth reason for using Agile is that Agile development is motivating and engaging. This post will expand upon this topic.

My position is that developers are knowledge workers, and as I pointed out in my article, “knowledge workers have the greatest understanding about their own work, and that they are the ones best qualified to plan and organize how they will accomplish that work.”

I’m not alone in this observation. In their book, High-Performing Self-Managed Work Teams by Dale E. Yeatts and Cloyd Hyten, Yeatts and Hyten stated, “Case studies have shown that the decisions made by SMWTs (Self-Managed Work Teams) are extremely effective because those making the decisions – the team members – are the most knowledgeable persons about the work.” (Buchholz, Roth & Hess, 1987; Ray & Bronstein, 1995).

Not only are those decisions effective, but allowing teams to operate autonomously – self-directed – is a great motivator. Why do so many people seek to start their own business? Money, yes, but another very common reason is autonomy. People want to live their lives the way they want to; they don’t want a boss telling them what to do, when to do it, and how to do it. The control factor present in most workplaces is a major factor in the decision to go it alone.

Tony Schwartz discussed this control factor in his book; The Way We’re Working Isn’t Working. Schwartz states, “The all-too common dynamic is today’s workplace is parent-child. Most employers tell employees when to come to work, when to leave, and how they’re expected to work when they’re at the office. Treated like children, many employees unconsciously adopt the role to which they’ve been consigned. Feeling disempowered and vulnerable, they lose the will and confidence to take real initiative or to think independently. Doing what they’re expected to do often becomes more important than doing what makes most sense.”

As a manager, I certainly want people thinking independently and taking initiative. I want people to do what makes sense, and by all means avoid following a procedure because “that’s the way we’re supposed to work” when that procedure doesn’t apply! I want people to understand our business goals and to think and act in accordance with those goals.

A great treatment on the subject of engagement and motivation is given by Dan Pink in his book Drive: The Surprising Truth about What Motivates Us. He relates a theory put forth by Edward Deci and Richard Ryan called “self-determination theory” which argues that we have three innate psychological needs – competence, autonomy, and relatedness – ingredients that Agile development endorses. Deci and Ryan feel that when these needs are met, we are happy, productive, and motivated. (No argument from me!)

In support of productivity gains through autonomy, Dan Pink cited a study conducted by researchers at Cornell University that examined 320 small businesses, half of which granted the workers autonomy, the other half relying on top-down direction. The results were fantastic! The businesses that offered autonomy grew at four times the rate of the control-oriented firms and had one-third the turnover.

While I'm on the subject of engagement and motivation, good morale is vital to positive motivation and engagement. Studies support that adopting Agile development definitely improves morale. The 2009 State of Agile Survey by VersionOne lists Improved Team Morale as the fourth-highest benefit obtained from adopting Agile development. Mike Cohn reports in his book, Succeeding with Agile: Software Development Using Scrum, that fifteen months after adopting Scrum, Salesforce.com surveyed its employees and found the 86 percent were having a “good time” or the “best time” working at the company. Prior to adopting Scrum, only 40 percent said the same thing – a solid improvement to say the least!

A couple of quotes that that capture the concept of autonomy, motivation and engagement:

“Mediocrity is expensive – and autonomy can be the antidote.” – Tom Kelley, General Manager of IDEO

“Traditional management is about compliance. If you want engagement, self-direction works better.” – Dan Pink

Finally, Agile development provides a sense of accomplishment that is very motivating. Through the delivery of working software early and often, there is a sense of meaningful accomplishment on a regular basis. There is also a feeling of belonging and relatedness due to the highly collaborative nature of Agile development.

I’ll summarize by quoting from my article. “This control over their workday, plus operating in a sustainable mode and the feeling of accomplishment that is a by-product of continuous delivery of working software, all combine to provide a solid motivational boost to each and every person on an Agile team.”

What is a manager to do? As Josh Leibner, Gershon Mader, Alan Weiss observed in their book The Power of Strategic Commitment: Achieving Extraordinary Results Through Total Alignment and Engagement, “Outstanding managers can create environments conducive to people motivating themselves.” Agile development creates this very environment – provided that you (as a manager) and your organization endorse and support it.

In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), my eighth reason for using Agile is that, Agile development expects continuous improvement. This post will expand upon this topic.

As I noted in my article, “Agile development expects its practitioners to be continually learning and adapting…” This concept is articulated in the final principle of the Agile Manifesto, which states:

“At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”

As a manager, I want to see everyone to improve and grow, and i feel that continual learning is vital in the software business. If you aren’t learning and growing (you’re “maintaining”), you are effectively losing ground. The world will move past you.

It is equally important that my development organization is utilizing the best practices possible to make individuals and teams as effective and productive as possible. Agile development actually makes a manager’s job easier with respect to learning and improving people and the software development process. You don’t have to explicitly create a “learning opportunity” for your staff (carving out time in busy schedules); the opportunity is created through the Agile process itself.

As I stated in my article, continuous improvement is enabled in part through sustainable development. Improving productivity requires that people grow and change, and in order to accomplish this, people need time to learn, think and reflect. Another component of improving productivity is getting people to accept change. The best way that I know of to accomplish this is to have them initiate the change on their own.

The other enabler is the individuals themselves. People must have the desire to actively learn and grow; they must value learning and be willing and able to act on what they learn, improving their personal and/or team productivity. By applying what they learn and then thinking and reflecting on how well it worked (and tweaking where necessary), new knowledge and expertise is developed. This is a true learning organization at work.

To facilitate this, Agile development incorporates the concept of slack. As James Shore and Shane Warden described it in their book, The Art of Agile Development:

“To keep up with their constantly expanding field, programmers must continually improve their skills. In doing so, they will often learn things that enhance their work on the project.

“Dedicated research time is an excellent way to encourage learning and add additional slack into your iterations. To introduce it, set aside half a day for each programmer to conduct self-directed research on a topic of his choice. Be completely hands-off. I recommend only two rules: don't spend this time on project stories or tasks, and don't modify any project code.”

I’ll make one point about the first sentence, which in part states, “…they will often learn things that enhance their work on the project.” Some people are better learners than others, and often times it is better to learn about specifics that are directly related to the work that is being performed or is about to be performed then something that you might need to know about in the future – there is direct applicability and reinforcement of the learning. It’s an application of the age-old, “use it or lose it” advice.

Team retrospectives are another way that Agile development fosters improvement. A retrospective is a meeting where a team looks back on their past iteration so that they can learn from their experience and apply this learning to future projects. In contrast, traditional, plan-driven projects normally hold something like a “post-mortem” at the conclusion of the project, when it is too late to improve that project.

I like this continuous improvement cycle of Agile development, not only because small improvements are being routinely made, but because the changes are automatically bought into by team members because they are the ones initiating the change. I also like regular retrospectives because they are also a great time to reflect on what went well, and what should be celebrated. Celebrating success helps keep people energized and engaged.

When it comes to improvement, some key points about retrospectives that I feel are important:

No blame. Retrospectives shouldn’t look to lay blame, but to examine – professionally and depersonalized – what needs improving.

Focus on what is in the team’s control. A team retrospective that focuses on exclusively that is outside of their control isn’t particularly useful to the team. Sure, maybe the team could have used an extra resource or two. The question is, did the team maximize the use of everyone? What or how could the current team have done to work more productively?

Drill down to actionable tasks. Don’t settle for high-level statements of a problem and/or vague notions of what could be done. Decompose the issue into actionable tasks that could be taken on in the next iteration.

Prioritize and select a task. A team doesn’t have to take on a suite of improvements all at once. In our organization, we use two-week iterations, so every two weeks there is an opportunity for improvement. Teams need to ask themselves, what would have the greatest impact on productivity for the team?

There are some managers who get uncomfortable with the concept of slack during an iteration and time dedicated to routine retrospectives. They don’t feel that they are maximizing productivity because team members aren’t engaged in “producing” software on a constant basis. This is counter-intuitive, in my opinion.

If you want productivity gains out of your knowledge workers – increasing the capacity of your existing workforce – then those people need to put knowledge into their knowledge banks. Knowledge workers cannot improve their capacity without investing time and effort to do so. This is a two-way street, however. The organization must recognize and provide the opportunity for this to happen, and the employees must take responsibility and take the initiative to follow through.

Agile development creates this opportunity. And it is up to the teams and the individuals on the teams to exploit the opportunity that is presented to them.

In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), my seventh reason for using Agile is that Agile development builds trust and relationships. This post will expand upon this topic.

My article highlighted two key areas where trust and relationships are built using Agile development:

• Between the team and the business (customers and/or stakeholders). 
• Between members of the team.

Building Trust with the Business
James Shore and Shane Warden explained this issue in their book, The Art of Agile Development:

“Many organizations I've worked with have struggled with an ‘us versus them’ attitude between customers and programmers. Customers often feel that programmers don't care enough about their needs and deadlines, some of which, if missed, could cost them their jobs. Programmers often feel forced into commitments that they can't meet, hurting their health and relationships.”

The business side has its drivers; sometimes they are the ones dealing with an imposed date – like meeting regulatory requirements. Other times, the business is responding to a competitive threat where delay is costly in very real terms, such as revenue and market share. And in other cases, there is an opportunity that the business wants to capitalize on to drive growth – a “first to market” scenario. Whatever the situation, the DATE that customers have in mind is very real.

On the software project side of the fence, other dynamics can come into play. In response to the business pressure, for example, a software project team can be driven into “optimizing” a tradition, plan-driven project before the project has even started. Why? Because the schedule – and indirectly, the team – isn’t trusted. Typically, it is management that is examining project schedules, and they are usually responding with one or more of the following motivations:

• A strong desire to complete the project as quickly as possible so that the team can move on to other, pressing projects.
• Pressure to meet the business needs within certain timeframes to win the business and drive revenue (or prove the worth of an in-house organization).
• A strong desire to ensure that the team is motivated by “just enough” schedule pressure to work up to their potential.

The reality is that everyone is concerned about whether the project team can deliver working software even close to the expected timeframe and anticipated budget. Documents and plans are fine, but it is the long gap between project inception and the first delivery of working software creates distrust with traditional, plan-driven projects.

Let’s face it, after months of dialog, reviews, and eventual sign-off to “get the requirements right,” customers are still going to feel uneasy, particularly in the early software construction phases when there is nothing to see. And despite everyone’s best efforts to fully comprehend the needs of the business, there will be questions during the development of specific features, and some of those questions will reveal changes – or even new features – that should have been a part of the requirements, but weren’t.

What can happen next is “the case of the imposed date” from a development point of view. The customer thought that they had expressed what they needed to development, and development provided an estimate. The problem is, as the team gets into the project, a deeper understanding occurs and the original estimates are in need of revision.

It is at this point when someone (usually a manager) reminds the team that they created the estimates – which are now considered cast in stone and not the approximation that they really were – and the team is held to its “commitment.” Doubt starts creeping in, and the question that is on everybody’s mind is: Will this project make it?

These days, it is no secret that a great many projects suffer schedule overruns. And when projects become challenged, conversations invariably begin about trimming features from the planned delivery or extending the delivery date, even before the customers have seen much – if anything – of the software that is under construction. At this point the customer’s fears are realized, and trust takes a hit.

If customers can see it, they can believe it – and ultimately trust those producing it. This is precisely the point that my article made, the benefit of trust that Agile development provides: “Through early and often delivery of working software to the business – even if this is nothing more than demonstrating the features – the business will grow to trust the development team. In addition, the continual dialog and the ability for the business to adjust and adapt the software gradually changes the ‘us versus them’ dynamic into a ‘we.’”

Referring back to The Art of Agile Development, James Shore and Shane Warden summed this up nicely: “Stakeholders don't know how to evaluate your process, but they can evaluate results.”

Customers don’t understand all of the ins and outs of software development, but they do measure us. Their ultimate measure is the end result – working software that either meets or fails to meet their needs. Their satisfaction can be – in part – a direct result of how much participation and input they experienced during the software project.

This is where Agile development lends itself to establishing trust and relationships. Agile development expects and embraces continual customer involvement with the development team. Instead of a large amount of time spent up front defining and refining all of the requirements, Agile development spreads this involvement out over time, accomplishing the same goal, but in different ways. Agile development also includes regular iteration reviews that demonstrate new features to the customer – a very visible display of progress that is much more convincing than reaching a milestone marker on a project plan.

This increased trust and improved relationship provides a couple of other benefits. If the customer wants add new features that affect the schedule, the trust and relationships that are built allow for candid conversations about what the customer will receive and when, including discussing trade-offs because new features are being added.

Even if the development team is struggling with a feature that is proving to be more complex than anticipated, the visible progress and constant contact with the team makes the “we’re struggling to deliver this feature” conversation is much easier to have. Because of the connected nature of the relationship, the customer has a greater appreciation of the effort involved.

The other benefit of trust and the frequent delivery of working software is that a significant amount of documentation is simply not required. In sequential projects, mounds of documentation exist to inform and guide development efforts because it is assumed that conversations will not occur. The requirements document is intended to capture all that there is to know prior to it being handed off to development. Continual conversations cut down on the need for comprehensive documentation; and contrary to what some contend, Agile development does NOT mean no documentation.

Building Trust between Team Members
The traditional, plan-driven approach to development organizes and optimizes its work functionally – with formal hand-offs between functions. People can start to identify with their area of specialty and not with the project or worse, not with the customer. The notion of specialists and optimizing work isn’t bad, and this model does work. Given that work centers around human beings, problems can and do arise.

When I first became a development manager, one problem that our organization had was that there was a wall between Development and QA. I worked hard with the QA manager and everyone involved, breaking those barriers down.

This involved coaching – casting QA as a benefit by focusing attention on the fact that our job was to prevent defects from getting out the door and into our customers’ hands. I began working on setting higher goals for our developers, asking more in terms of design, code reviews, unit testing, and the like. The QA manager worked at raising the QA game and pointing out that developers weren’t being sloppy just because bugs were showing up – at least not if they were using sound practices like conducting design and code reviews.

We also held project meetings that fostered and encouraged more open dialog. Eventually, everyone began to gain an appreciation for the skill and work required on both sides of the fence, and despite that fact that at the time we were still a plan-driven, functionally-oriented shop, the teamwork and rapport significantly improved between the Development and QA organizations.

Agile development takes building teamwork and rapport to a higher level. Project work matters more than a functional role, and as I noted in my article, “Agile teams make the most effective use of the team members as dictated by the needs of the team to meet its commitments. The shared goal becomes more important than each individual working strictly within his or her area of expertise, with defined policies and procedures for handing off work between functional roles. This breaks down barriers between functional disciplines, enabling the team to reach higher levels of productivity.”

This means that a developer can and should help test (particularly if that testing involves creating automated tests) to help the team finish its work. Meeting an iteration commitment shouldn’t be an individualized, “I’ve completed my tasks,” but a, “How is the team doing in meeting our commitments?” With Agile development, there is a strong expectation that the work is all about the team getting across the finish line together.

Because everyone is working closely together – preferably co-located – and focused on completing the team’s highest-priority items first, trust is established through continual interaction and delivery. The daily stand-up meeting reinforces individual commitments to accomplishing meaningful work each and every day, and the expectation that Agile development places on continuous team and individual improvement builds trust as team members and teams improve their game.

In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), my sixth reason for using Agile is that Agile development enables emergent innovation. This post will expand upon this topic.

It is no secret that companies desire greater innovation to differentiate themselves in today's marketplace. Emergent innovation is all about deriving innovation organically from within the organization as a natural course of performing day-to-day work.

I’ll admit that this particular reason for using Agile development was of my own choosing, and I cannot point to specific studies that support this as a benefit. Since I was responsible for writing the article, I wanted to give my opinion on a benefit that I feel should be a part of Agile development.

Step One in enabling emergent innovation is to provide a sustainable development environment. As I noted in my article, “Teams that are working constant overtime to meet schedules simply lack the time or inclination to think about anything else other than the difficult schedule in front of them. In sustainable development environments, people have the time to think more about the business and explore – creating the potential for innovation that did not exist previously.”

Step Two is to have conversations as a team. Requirements (User Stories) should not be considered cast in concrete, but should serve as a basis for a conversation about what the business wants and the anticipated benefit(s) the business expects as a result. As I noted in the article, “…the dialog between business experts and the technical experts can yield unexpected results, like turning complex, difficult features into elegant, differentiating features.”

We’ve experienced this. We’ve had teams review User Stories and discuss how certain features could be implemented – a back-and-forth conversation about the business workflow and what was realistic for software to handle automatically along with how the user needed to interact with the software to accomplish his or her goal. We’ve been able to eliminate unnecessary steps and simplify the software while still providing a great user experience that met the needs of the business.

Sometimes, users are asking for features that are based on other software that they have experience with. The trick is they don’t always know how to separate how they perform tasks today with what is possible tomorrow. This is why understanding the anticipated benefit is an important component of the User Story, and why conversations between those who understand the business and those who understand software construction are highly beneficial. You can end up i a much better place than either side could have imagined by working in a sequential, isolated hand-off development approach.

Emergent innovation can also involve an entire product. We initiated one project that was little more than a vague notion of the actual problem that we wanted to tackle. This was a classic R&D effort where we gradually – and experimentally – tackled the issue of exposing complex, back-end insurance processing to web portals.

We initially felt that we needed to build a portal itself, but as the team immersed itself into the effort, a shift occurred. The result was a middleware set of tools and technologies that enable us to interrogate a back-end system and generate schema and meta-data that describe the workings of the back-end system. (And an even better, recognition for the developers in the form of a patent application.)

The result is that we don’t create web portals, we allow the definition and management of templates, rules, and workflows that govern the semantic display and behavior of information that can be contained within web portals. We make it easy to access back-end systems and propagate changes out to the web, with our middleware facilitating the dialog between the web portal and the back-end system(s). Our customers have full control over the look and feel of their web portals, leveraging our technology to make their lives a lot easier.

Naturally, this type effort required some of our best people. People who have a proven track record of creative thinking, the ability to deal with ambiguity, and that inquisitive, exploratory mindset that enabled them to define the goal and then work towards achieving it.

What are your thoughts? Does emergent innovation belong among the Top 10 Reasons for Using Agile?

In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), my fifth reason for using Agile development is that Agile development creates a sustainable development environment. This post will expand upon this topic.

As a software development manager, my responsibility is to make sure that our development organization produces working software day in and day out, year after year. And like many other managers out there, it seems that every year I’m being asked to do more with less. I can accomplish this in one of two ways:

1. Squeeze my staff for all they are worth (and replace them out when they burn out).
2. Work smarter.

I personally like creating a collaborative, fun, sustainable work environment that is focused on retaining good employees and not working them into the ground to meet imposed, aggressive (ridiculous) deadlines that are based on little more than hopes and dreams. Call me crazy, but I like treating professional knowledge workers as professionals. And I enjoy seeing people develop and grow – it’s truly rewarding to see people reach new heights!

Sustainable software development is all about meeting the needs of the business on an ongoing basis, which is typically translated into meeting project schedules. There are other ways of meeting the needs of the business, which I discussed in Agile Development Embraces Business Agility. The issue of software project schedules and the usual overtime response in order to meet those schedules is worth examining in term of productivity and results.

In my article I stated that, “Software projects rarely meet their initial schedule. Overtime with the mandate to ‘work harder’ is frequently used as it becomes apparent that a project will be running longer than anticipated.”

This has been an all too-common experience for many software professionals, and a great deal of anecdotal evidence and various studies that support that schedule overruns are frequent:

In a Dynamic Markets Limited Independent Market Research Report commissioned by Tara Consultancy Services in August 2007, 62 percent of the IT projects failed to meet their anticipated project schedule.

The latest Standish Group report in 2009 report shows that only 32 percent of IT projects were considered successful – meeting the schedule, budget, and delivered with the expected features.

In his book, Software Runaways: Lessons Learned from Massive Software Project Failures, Robert Glass found that schedule overruns were very common, pegging schedule overruns at 89 percent. (The book profiled 16 of the largest and most publicized software failures of the 1990s.)

Using overtime as a mechanism to deal with software project schedule issues creates far more problems than it solves, and generally inhibits the creation of a viable, productive, long-term organization. The longer-term risks of burnout, turnover, and overall sustainability are commonly acknowledged (and frequently ignored), but the short-term effects are rarely mentioned.

In the first place, the productivity of knowledge workers (like programmers) plunges with heavy doses of overtime. Research from The Journal of American Epidemiology – studying people who work greater the 55 hours – found that individuals had problems with memory, problems with functioning appropriately and problems with reading and comprehension.

From a software development standpoint, a major short-term issue associated directly with overtime is quality. In a paper, Impact of Overtime and Stress on Software Quality by Balaji Akula & James Cusick, Baleji and James focused on the impact of project teams working on an aggressive schedule, studying four projects over a two-year period (three of which had an aggressive schedule).

The chart below demonstrates a dramatic difference in defect rates when no overtime is involved:


Project #1 Project #2 Project #3 Project #4 Estimated person hours 784 416 416 528 Overtime person hours 175 167 0 198 Defects 2038 1545 142 2000
I’m quite sure that I know why defect rates are low when no overtime is present; I’ve experienced this many times as both a programmer and as a manager. Unrealistic schedule pressure and overtime not only introduces fatigue that produces more defects, programmers stop thinking as deeply and carefully about their work in response to the pressure to meet the deadline. Corner-cutting begins to creep in at every turn as output becomes a major focus – at the expense of practices that help to produce more reliable code.

Overtime has other problems as well. Since programmers are knowledge workers, they need to putting knowledge into their “knowledge banks,” something that will pay dividends later. Routine overtime on routine project tasks eliminates the time for the professional growth that will be of future benefit. (And in other cases, people are doing themselves a great disservice by failing to continually acquire new knowledge, as Pawel Brodzinski pointed out in a post, People Don’t Want to Learn.)

Then there is the whole work/life balance issue. My wife, Lauri-Ann, has referred to herself as a “software widow” at times when I’ve been very involved – and I mean working some very long hours – on software projects. You can tell when you are tipping the scales in favor of work too much when your wife pleads with you to put the computer aside to watch your daughter play soccer on a Saturday morning. (True story, Lauri-Ann almost had to hit me over the head.) In the long run, this isn’t healthy…

Sustainable Development
I support Agile development because it promotes sustainable development as one of its 12 Principles, as found in the Principles behind the Agile Manifesto. The principle states, “Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.”

Agile development doesn't try to predict the future by estimates alone, let alone try to make that future happen via overtime. With Agile development, the future can be predicted based on actual team velocity. Agile teams size User Stories in points and tracks team velocity – the User Story points that are completed in each 2-4 week iteration.

Stories are considered done when they are delivered as working software with high quality. Over a period of time, it becomes crystal clear how much work can be realistically accomplished by a team on a sustainable basis. Future delivery can be predicted by contrasting the team velocity against the active product backlog and the size of the User Stories contained in that backlog.

Once you have a clear picture of what is realistic and sustainable, how should you seek to improve team performance? It shouldn’t be about working harder, it should be about working smarter. How is the collaboration? Are there personnel issues or friction? Are practices such as Test-Driven Development being utilized? I’ve witnesses Agile teams working very productively, accomplishing more than they were when they were working overtime through the proper implementation of Agile development. The output was higher, it was sustainable, and morale was better.

When it comes to having developers maintain their skills and working smarter, one contribution that Agile development brings to the table (if you choose to use it) is the concept of slack as a technique to provide for research time each iteration. I first encountered this concept in the book, The Art of Agile Development by James Shore and Shane Warden. They state:

"To keep up with their constantly expanding field, programmers must continually improve their skills. In doing so, they will often learn things that enhance their work on the project.

"Dedicated research time is an excellent way to encourage learning and add additional slack into your iterations. To introduce it, set aside half a day for each programmer to conduct self-directed research on a topic of his choice. Be completely hands-off. I recommend only two rules: don't spend this time on project stories or tasks, and don't modify any project code."

What about teams meeting their commitments? Shouldn't they work overtime to meet what they committed to? If you have a team that struggles to meet its commitments every now and then, you actually have a great situation! It means that they are taking on work that tests and stretches them. It should be a judgement call whether to roll the work into the next iteration or deal with it now. It really depends upon the urgency and circumstances.

If there is a critical customer deliverable with a tight timeframe, a good team will recognize that and get the job done, and they will set aside their research time to make things happen. And yes, there are times that as a manager I've had to remind teams that a customer was waiting on us, and that yes, working over the weekend was necessary. There is nothing wrong with a spike of activity from time to time.

There are other circumstances when it is OK to let the work slide. Software development isn't highly predictable, and what appears simple when it is estimated can turn out to be more difficult in the actual implementation. In this case, why punish people by mandating overtime? Since it is a difficult problem, taking the extra time to deal with it – keeping people fresh at the same time – will improve your odds that it will be dealt with correctly the first time.

I've seen comments on LinkedIn and on other blogs that lead me to believe that there are some very wrong implementations of Agile development out there. Instead of sustainable development, the term "Agile" is being misused and abused to drive a series of short-term, "mini death marches" where the goal is to code quickly (without thinking) and to drive the output of the team in unrealistic and unhealthy ways. Unfortunately, this is the very issue that Agile development seeks to overcome.

Don’t be misled, though. Agile development takes discipline. It takes knowledgeable, capable, engaged people. It takes a supportive management team. It takes training and time. It is work and it is change, but in the end I believe that it is worth it. Regular use of overtime should be viewed as a serious problem.

In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), my fourth reason for using Agile development is that Agile development increases productivity. This post will expand upon this topic.

Before I begin, let me state that I understand that traditional projects can be successful and productive. Many people point out – rightly so – that many aspects of Agile development are not new. However, the question of productivity as it relates to Agile development is worth exploring. Even if you don’t want to be an Agile shop, perhaps there are aspects that you can adopt that will improve your situation…

Surveys and studies have attempted to determine to determine if Agile development is improving– or least providing a strong perception of improving – productivity. While everyone is not reporting productivity improvements, a high percent certainly are. Let’s take a quick look at some percentages:

82 percent reported a “somewhat higher or much higher productivity improvement” in Dr. Dobb’s February 2008 Agile Adoption Rate Survey Results.

73 percent reported “improved or significantly improved” productivity in VersionOne’s 2009 State of Agile Survey.

83 percent reported an improvement in productivity at Lockheed Martin, as reported at the Agile Vancouver: Much Ado About Agile, Sep 2, 2007 by Mike Zwicker in a presentation titled War Stories: Fighter Jets and Agile Development at Lockheed Martin.

Reaching further back, 93 percent stated that productivity was “better or significantly better” in the Shine Technologies 2003 Survey.

How is this happening?

Practices
Scrum is taking the lead as the Agile method of choice, but there is growing acknowledgement that a Scrum/XP hybrid provides more along the lines of total productivity improvement. My article focused attention primarily on the autonomous, teamwork orientation that Scrum provides – although I admit that I should have mentioned something about technical practices.

Some key practices of Agile teams employ that improve productivity:

Automated Testing: Ideally, these are small tests that are capable of being executed many times a day and can confirm that a newly-developed feature is working correctly as well as confirming that pre-existing features are continuing to function as expected.

Automated Unit Tests/Test-Driven Development (TDD): Agile development insists upon developer unit testing, and the best way to accomplish this is through automated testing. TDD is a “test first” approach where a unit test is written before the code is written; the test fails and the developer must code to make the test pass. This approach has the advantage of forcing developers to think more about their interfaces, helping to make the software more understandable and self-documenting.

We’ve been increasing our attention on automated unit tests, and the first benefit we obtained was that our developers were delighted that they could re-factor with confidence. And QA was more than grateful that automated tests quickly and easily proved that the behavior of the software did not change, with no manual testing required on their part. In the end, our velocity improved, allowing us to develop additional features that we could not have completed without the initial investment to build our suite of automated unit tests. (This investment was made gradually as we built out new functionality – although we did need to invest time in creating tests for code that was already created without automated unit tests.)

Code Reviews/Pair Programming: The benefits of code reviews have been known to the industry for many years; equally known is that the earlier a defect is detected, the cheaper it is to correct. Catching coding problems is best accomplished by a code review. Pair programming is one way to accomplish a real-time code review.

A recent Forrester study, The Value and Importance of Code Reviews, published in March, 2010 (commissioned by Klocwork) found that code reviews are considered to be of high value by those who conduct code reviews. The benefits cited:

• Bugs are uncovered earlier in the life cycle.
• Best practices are shared.
• Refactoring and code simplification are encouraged.
• Reduction in wasted time (later in the project).

There is a way to combine the development of automated unit tests and code reviews. We have not tried this out in our office (yet), but I did attend a recent training course where we conducted a side session that combined Test-Driven Development and Pair Programming. One programmer wrote the test, and the other programmer implemented the code. There was continual dialog about the business feature and the design and implementation as we worked through the example. The end result was a shared understanding of the code, solid, simpler code (we simplified the instructor's sample code), and an automated unit test. Does anyone have real-world experience with this technique that they would like to share on a guest post?

Continuous Integration: Instead of keeping portions of a system separate until the end of a project, Agile projects integrate and deliver working software early and continuously. This avoids problems of late-stage integration that some projects suffer, and helps to build a greater degree of quality into the system through the testing of features through the architectural layers.

Insistence on Quality: Agile development has a philosophy that quality comes first. Features should not be considered done until they are of high quality; defect-fixing is not an activity that occurs late in the project cycle. In the event that a defect is found with a feature that was declared as done, work on a lower-priority feature should be deferred in favor of making sure that the higher-priority feature is of high quality.

Teamwork
As I noted, my article focused on the team-oriented aspects of how Agile improves productivity. I made the statement that, “Agile development focuses the team's attention like a laser on delivering the highest-priority, highest-valued features in short increments of time using the most productive mechanisms to accomplish the work.”

Part of the “most productive mechanisms” are the practices noted above. Focusing on the highest-priority, highest-valued features is just that – a concentrated, team-based effort. We’ve found through our own experiences that teams should limit their work in process to maximize productivity.

We use two-week sprints our organization, and teams make commitments to accomplish certain work within that two-week period. It is tempting for teams to start as much of that work as early as possible, but I advise caution. There can be side-effects:

1. Less collaboration occurs, and you end up with a team that is a collection of individuals working on their own tasks, leading to point number 2.
2. Teams may end a sprint with a lot of work in process, but nothing actually completed.

Limiting work in process forces teams to truly discuss the highest priority items and finding ways to collaborate and truly share and own the work as a team. You might get pushback on this, as people tend to react with “there isn’t anything that we can all work on,” but there is usually ways in which people can collaborate.

We’ve also experienced the benefits of establishing a truly collaborative team environment. We had one team that did nothing other than change how they worked. We had a classic case of a “team” that was simply a collection of individuals working individually on their tasks. There was little to no collaboration, functional hand-offs, and plenty of overtime.

We started over by grooming the backlog and discussing – really discussing – each feature. We found that some features were vague, and others were unnecessarily complex based on the goals that the business had in mind. By having a real dialog with the Product Owner, we were able to make changes that met the business needs more effectively, and in a way that was simpler to implement from a development standpoint.

The day-to-day work of the team changed as well. Everyone tasked out each User Story, discussed the validations, and collectively estimated the work (including discussing any deviations in the estimates). And during the course of implementing a User Story, the team collectively discussed the impact of any change – yes, issues surfaced as the team got into the details of some features. But there was a collective understanding and full team ownership of each and every feature.

The result? Without adding or changing any technical practices, focusing solely on teamwork and building true collaboration, the team achieved a little over 30 percent increase in productivity. Our measure was based on the team’s estimate of when they would complete the work based on their old approach and velocity. We simply beat it, and this included eliminating the overtime that the team was working.

This brings me to another point that I made in the article. Productivity increase comes from teams operating autonomously, where the teams make informed decisions about their day-to-day work without the need for constant managerial direction. Not only does this expedite certain decisions by keeping them localized, employees who have more control over their day-to-day work are more energized and engaged about their work.

A final note about improving productivity through teamwork that I did not mention in my article (one that we’ve also proven in our experience), is that small teams are more productive. This has also been proven in other studies, such as those made by Quantitative Software Management (QSM), a company that maintains a comprehensive database of software development project metrics.

Here’s a pop quiz for you, based on QSM data: There are two software teams that must deliver a project with a size that is estimated to be about 40,000 lines of code. One team, Team Large, has 29 resources compared to Team Small, with 3 resources. How much faster do you think Team Large will be?

According to the QSM data, the difference would be a mere 12 calendar days! The other significant differences would be the fact that Team Large would consume 151 more person-months more Team Small, and Team Large (on average) would create six times as many defects! The additional lines of communication and overhead associated with large project teams create more problems than are solved.

Hyper-Productivity
Jeff Sutherland coined the term Hyper-Productivity, defining it as a 400 percent higher velocity improvement. It’s an impressive marketing term, but does Hyper-Productivity exist?

It’s possible, based on studies. In his article, An analysis of XP, TDD, Pair Programming, and Scrum (Using Real Options), Dr. David F. Rico arrived at the following figures:


Low Median High Productivity Improvement 14% 122% 712% Quality Improvement 10% 70% 1,000% Schedule Improvement 11% 71% 19%
I don’t think that everyone can improve by 400 percent or more; my personal opinion is that this depends entirely on where you start from. If you have a good sense of software development and you are following a sound methodology that includes good technical practices, I still feel that you can improve, but not to a dramatic degree that places you in the Hyper-Productivity category. What are your thoughts about Hyper-Productivity?

In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), my third reason for using Agile development is that Agile development reduces risk. This post examines the topic in greater detail.

Every software project has risk associated with it, ranging from business, project, and technical risks. I’m sure that a variety of risks come to mind, but the key points are:

• There isn’t a software development project that doesn’t face risk.
• Reducing risk involves mitigation tactics on multiple fronts. 

The first paragraph in my article under reducing risk stated one problem with software development is the “delivery of software where the users ‘got what they asked for, but it isn’t what they wanted’ because the requirements were not understood in the first place.” I’ve experienced this myself more than once, and others that I’ve talked to over the years have the same experience.

Requirements problems account for approximately 30-40 percent of a software development budget, based on figures by Dean Leffingwell (reported in his book, Managing Software Requirements, 2003). This includes the experience that Salesforce.com reported at the 2007 Agile Conference, where Steve Greene and Chris Fry noted that Salesforce.com “…experienced problems that most plan-driven projects experience, such as late feedback on features at the end of the release cycle...”

Late feedback at the end of the release cycle isn't bad, unless the feedback is about what you don't have right. When project teams are faced with enough of these requirements problems at the end of a project, discussions typically begin to either push out the project date or to drop features in order to meet the date.

Using an Agile development approach, on the other hand, significantly reduces the risk of this problem from occurring. Because there is continuous involvement from the business, there is less opportunity for misunderstandings to occur. And because of frequent delivery and inspection, a requirements problem will be detected early in the project, allowing the business to weigh the cost and prioritize a requirements change against other features.

Even if a new requirement emerges (a.k.a., scope creep), Agile development accommodates this through rigorous prioritization of the features based on value. Everything can’t be a priority, but if a new feature is determined to be important and valuable mid-way through a project, Agile development allows the feature to be prioritized and worked – with lower-priority items moving down in the queue (backlog).

Controlling scope creep itself if managed through continuous involvement and dialog. The benefit of collaborative development and managing scope creep was written about in an article by Capers Jones in 1996 (Strategies for Managing Requirements Creep), where joint development between user representatives and development was noted to cut requirements creep nearly in half. It is interesting to note that Capers Jones was reporting this in context of Joint Application Development that has been around since the 1970s. While the concept of collaborative development is not new, but Agile development exploits it.

Another way to mitigate risk is to improve project visibility. Understanding exactly what challenges and obstacles that a project team needs to overcome, plus getting a realistic appraisal of their true progress reduces risk because it become possible to intervene to help the team. In the 2009 State of Agile Development Survey sponsored by VersionOne, the second-highest benefit obtained from implementing Agile development was improved project visibility. 83% of respondents said implementing Agile showed either improved or significantly improved project visibility.

We’ve used increased visibility to mitigate risk at my own company. Those of us on the management team formally review each project weekly with the Product Owners and Scrum Masters. We also make every effort to attend the daily stand-ups. This way, we can get a good sense for what is going smoothly and what is not – and we can take action to do whatever needs to be done to help the team move forward.

This counts for the smaller, day-to-day problems and the larger, “we’re not going to make when we thought we were” problems. For example, we encountered one problem where it was apparent that the team was simply too big that is was literally tripping over itself, so we split the team in two and assigned Agile coaches to each team and improved our velocity as a result.

As I noted in my article, the concept of delivering working software can keep schedules and budgets in check, because the software is usable from the outset. If the project begins to take more time than planned, the business has the business has the option of discontinuing making further investment and using what has been delivered to date. Cutting an Agile project short doesn’t mean complete failure. You have some usable software, if you choose to use it.

Again, because Agile development delivers early and often, it gets around another all too common problem: the last 20% of the project takes as long or longer to complete as the first 80% of the project. A number of reasons can cause this: poor up-front estimation, late-stage integration issues as the project team attempts to “bring everything together,” requirements problems, quality issues (because a suite of testing was waiting for a code complete phase), deployment problems, etc. Here’s a couple of links to those who talk about the last 20% occupying large amounts of project time:

http://www.sqlmag.com/article/sql-server/the-80-20-rule.aspx

http://stackoverflow.com/questions/608748/how-to-avoid-the-80-20-rule-in-software-development

In terms of studies that that specifically quantify the risk reduction component of Agile. The 2008 and 2009 State of Agile Development surveys sponsored by VersionOne, contains an actual risk reduction category. 65 percent reported a significant improved or improved risk reduction in project risk, which was consistent with the 2008 figure of 64 percent.

There wasn’t a specific explanation in the studies about what risk covered, but another report, Agile Development: Results Delivered (by VersionOne, 2007), based on their 2006 State of Agile Development survey notes the following about risk:

“Agile development provides empirical control processes required when precision is needed and complexity or change is at a high level. Regular assessments through daily meetings and iteration reviews provide visibility throughout a projects lifetime. As a result, risk is reduced.

“Because planning occurs every iteration, the project risk falls because teams see actual results and reprioritize work to align with the business goals on an ongoing basis. Agile projects can dramatically reduce risk through delivering working software that meets the business goals faster. In contrast, with traditional development processes, risk remains high throughout the development process and is not significantly reduced until project completion.”

In general, the continual delivery of working software and the highly collaborative, visible nature of Agile development reduces many of the risks that traditional, plan-driven projects face. Agile development may not always eliminate the risk, but it should allow you to detect problems earlier, allowing you to and the team to adapt.

In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), my second reason for using Agile development is that Business agility is embraced. This post will expand on the topic.

Wikipedia defines business agility as: “The ability of a business to adapt rapidly and cost efficiently in response to changes in the business environment.”

Given today’s global, competitive economy, the ability for the business to respond to change is critical. Does it feel that change is continuing to accelerate to you? If so, at what rate?

If you want an extreme take on the subject, Ray Kurzweil states in an article, Understanding the Accelerating Rate of Change that, “The whole 20th century, because we’ve been speeding up to this point, is equivalent to 20 years of progress at today’s rate of progress.”

Sometimes, the consequences of failing to adapt is failure of the business. Faisal Hoque makes the case for business Agility in an article, The Speed of Business Today, asserting, “Constant change is the new dynamic of the global economy, and makes agility even more necessary than at any point in business history.” Faisal then states:

• Only 74 of the original 500 companies in the S&P Index are still on the list 40 years later―a mortality rate of more than 10 per year. 
• The average life span of an S&P 500 company has steadily decreased from more than 50 years to fewer than 25. 
• Projecting forward, it’s likely that only about one-third of today’s major corporations will survive as significant businesses for the next quarter century. 

I think that it is safe to assume that technological progress is due to slow at some point in time, but none of us really know when. Relating business agility back to software development, the observation that I made in my article was simple: “Software development practices shouldn't run counter to business needs by forcing the business to choose and adhere to a set of pre-determined features that will be delivered months, if not years, into the future.”

This is contrasting Agile development against a traditional, plan-driven software development approach. As the rate of business change continues to increase, if only seems prudent to utilize a software development approach that mirrors the needs of the business, particularly since the purpose of software development is to support the business. Agile development welcomes and adapts to change, which means that Agile software teams can adapt in accordance with continually changing business conditions.

My brother Brian made an interesting observation about software projects at his company (he works for a large company that I won’t name here). They struggle with containing costs and overall project throughput because they still use a plan-driven approach and they want everything “done right.” The problem is, doing things right in their context means adding features up front so that they “don’t have to come back to the project later.”

In an effort to nail every project and then move on, my brother’s company is costing themselves time and money. The notion of getting in and out quickly – failing fast in some cases – is not present. They COULD be introducing another problem for themselves as well. (If they aren't, others do have this problem.) In their attempts to be thorough, could be spending some of their valuable time building out features that they don’t really need.

Does this sound like your situation? If so, take a look at a widely-quoted study by the Standish Group, a typical system has a variety of features/functions that are rarely or never used:


The message here is simple: What you think you need at the outset of a project may not always be what you really need. Large, plan-driven projects with a large feature count will increasingly be at odds with the need for adaptability on the business side. A smaller project with a well-defined scope and feature set where you “get in and get out quickly” certainly helps, as does a methodology that supports adaptability.

Using an Agile development approach, the most valuable, prioritized features will be delivered in short periods of time with continuous involvement from the business. The business gets to use the software and provide feedback, both in terms of the actual features delivered and about what the future priorities should be. Agile development projects are thus able to easily adapt in accordance with changing business conditions.

In my own company, we frequently review priorities as a management team, and we’ve certainly seen our priorities change throughout the year. As time marches on, information becomes available, information that you did not or simply could not have at the beginning of the year. Did those prospects purchase our software? Did some new competitive threat emerge? In addition, getting software into the hands of the users as early as possible yields valuable feedback and can help identify new, valuable requirements, something that we’ve experienced.

Other companies value this capability as well. In the 2009 State of Agile Development Survey sponsored by VersionOne, the number one benefit obtained from implementing Agile development was the ability to manage changing priorities. 90% of respondents said implementing Agile either improved or significantly improved their ability to manage changing priorities. (The survey data included 2,570 participants in 88 countries.)

An older survey, the Shine Technologies 2003 Survey, was another global survey of actual experiences using Agile development. The highest ranked positive feature reported in the survey was the ability to “Respond to change over plan,” which came in a 47.3%.

Another great, real-world example of improving business agility is the story of Salesforce.com. Salesforce.com was growing rapidly, but it was becoming increasingly difficult for them to deliver. Salesforce.com experienced problems that most plan-driven projects experience, such as late feedback on features at the end of the release cycle, and long, unpredictable release schedules. The frequency of Salesforce.com’s releases went from 4 per year to 1, meaning that customers were waiting longer to get less from the company. This is far from embracing business agility!

Salesforce.com undertook an all-in, major transformation to Agile development. Within one year, Salesforce.com delivered 94% more features, and Salesforce.com calculates that it delivered 500% more value to its customers compared to the previous year. And within 2 years, Salesforce.com’s revenue had doubled. (These specific numbers are reported in Mike Cohn’s book, Succeeding with Agile: Software Development Using Scrum.)

I’ve embedded the complete Salesforce.com presentation by Steve Greene and Chris Fry that talks about Salesforce’s Agile transformation given at the 2007 Agile Conference so that you can view Salesoforce.com's story for yourself.
Salesforce.com Agile Transformation - Agile 2007 Conference
View more presentations from Steve Greene.
While traditional, plan-driven projects have certainly succeeded (and I’ve been a part of them), what are your thoughts on meeting the competitive demands of today’s world? Is software development adaptive and response enough? Is Agile development the right solution to this problem?

Introduction
In my post, What is Better About Agile Development? (a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com), I listed Superior ROI as the first reason for using Agile development. This post will explore the topic in greater detail.

I set a condition in my article that Agile projects provide a superior return on investment compared for other software projects where “the business is forced to wait for the completion of the entire project before it can begin deriving a benefit from that investment.” I’m therefore comparing traditional, plan-driven projects to Agile.

I recognize that there are hybrid solutions in use, but I’m using these comparisons to highlight the benefit of Agile development against traditional development in the purest sense – to examine what is working that we should all be considering, regardless of the development methodology in use.

Just to keep definitions clear, a traditional, plan-driven approach is one that operates with formalized phases of phases of requirements, design, construction, testing, and maintenance, completing each phase before moving to the next. These types of projects strive to be predictive. Conversely, Agile development is an umbrella term that encompasses several actual methods (XP, Scrum, etc.), but in general Agile projects are incremental and adaptive.

And the Survey Says...
Leading off, Scott Ambler’s December 2008 Software Development Project Success Survey specifically asked respondents about their ROI and included this in the results. Scott used a formula to derive an effectiveness score, and Agile development’s ROI score was 3.9 versus 0.2 for traditional development on a scale of -10 to +10.

In addition, a detailed treatment of various studies has been performed by David F. Rico in a couple of articles, and he includes a long list of references (and commonly cited sources) that serve as a handy reference if you want to examine a variety of Agile development studies for yourself.

In his article, What is the Return on Investment (ROI) of Agile Methods?, David Rico’s stated purpose was “to examine and identify the ROI of agile methods. More specifically, its purpose was to identify the ROI of using agile methods in their entirety, not just some of the tools and techniques of agile methods like pair programming.”

His conclusion: “The benefits of using agile methods range from 10% to 100% for increased cost-effectiveness, productivity, quality, cycle-time reduction, and customer satisfaction. The use of agile methods as a new product development approach does result in increased ROI. This begins to dispel the notion that agile methods result in lower ROI than traditional methods.”

In another article, What is the ROI of Agile vs. Traditional Methods? An Analysis of XP, TDD, Pair Programming, and Scrum (Using Real Options), David Rico’s goal was “examine scholarly studies of Agile Methods and survey the range of quantitative costs and benefits associated with the use of Agile Methods. Data were compared to costs and benefits of Traditional Methods such as CMMI®.”

David concluded that, “On average, studies of Agile Methods reported 29% better cost, 91% better schedule, 97% better productivity, 50% better quality, 400% better satisfaction, and 470% better ROI than CMMI®.”

He also made another good observation: “The latest trend is to mix-and-match Scrum and XP to tap into practices like Pair Programming (PP) and Test-Driven Development (TDD) to increase productivity and quality.” One Agile method alone is not the recipe for success, and like a lot of things, we need to look under the covers to discover what is really working. Some of these are practices that can be employed on any project.

The Numbers...
In terms of working the numbers, John Scumniotales has a nice post, Why incremental development is better - An ROI perspective, that demonstrates how delivering early and often improves ROI. John also provides an ROI/NPV Excel spreadsheet calculator that can be downloaded so you can play with the numbers for yourself, as your mileage may vary. John’s sample project contrasts a traditional (non-incremental) approach versus an Agile (incremental) approach:


Dennis Stevens made a solid observation in the comments section of John’s post, stating, “Some features are more valuable than other features… Higher earlier benefit actually creates a bigger spread.”

These comments address another observation that I made in my article and something that is important for improving results: “Agile does ask for discipline and participation from the business as part of the deal, such as rigorously prioritizing the features…” and, “In return, the business gets its highest-valued features delivered early…”

Our Experience...
We’ve experienced superior return at my own company as we’ve transitioned to Agile. (My posts don’t delve too far into the specifics of my own company to avoid any entanglements with competitive disclosure or breeching of confidences.)

In the past, we’ve used the traditional plan-driven development approach. We performed the classic requirements elicitation, refinement and development, and then moved on to design and build out the architecture in its entirety prior to plugging in the business features. I’ve been involved with projects that have required – based on the defined feature set and staffing – over one year to complete before we could release and begin generating a return.

In comparison, we initiated the development of a brand new product after our adoption of Agile development. This product was released much earlier, allowing us to begin generating returns much sooner than would have been possible with a traditional approach. In fact, we delivered this product to one customer before our first, official release. We’ve benefited greatly from an early return on investment along with obtaining early feedback.

The feedback that our customer provided us allowed us to make adjustments to features that we thought we had right, but didn’t. We thus obtained two benefits in one: we generated a return very early in the product cycle, and our official product release was stronger as a result.

We’ve also taken advantage of another aspect of Agile development: adaptability. We appeared to have a solid feature set defined, but feedback from customers have led us to adding new, compelling features and re-prioritizing our product backlog to work on those valuable features now. We aren’t measuring ourselves against “planning the work and working the plan”; instead, we’re adapting and responding to the needs of the market.

How Does Your Project Portfolio Look?
ROI from a single project is one thing, but many companies have a portfolio of projects, and it is worth considering what the return of the entire project portfolio. A failed software project obviously provides no return and can easily drain significant capital from a company, placing a greater emphasis and need for other projects to succeed.

Every year, the much quoted Standish Group Chaos Survey reports outright project failure or “challenged” projects – projects that fail to meet the classic definition of success in one or more ways, such as incurring a budget or schedule overrun, or having features trimmed from the final delivery, as noted in the table below:

2009 2006 2004 2002 2000 1998 1996 1994 Successful 32% 35% 29% 34% 28% 26% 27% 16% Challenged 44% 46% 53% 51% 49% 46% 33% 53% Failed 24% 19% 18% 15% 23% 28% 40% 31%
The Chaos Survey paints a dismal picture of the software industry, but it is worth noting that the very definition of software project success can and is disputed, as demonstrated in Scott Ambler’s December 2008 Software Development Project Success Survey. There were 279 respondents to Scott’s survey, and their definition of success in comparison to the classic definition is as follows:

• On Time: 58% believe that delivering when the system is ready to be shipped is more important than delivering on schedule.
• On Budget: 70% believe that providing the best ROI is more important than delivering under budget.
• Of High Quality: 82% believe that delivering high quality is more important than delivering on time and on budget.
• Delivered with the Expected Features: 83% believe that meeting actual needs of stakeholders is more important than building the system to specification.

Given that definition of success is disputable, it is certainly in the best interest of everyone to have a candid conversation about the definition of success before taking on a project! But the main point is that one disputes that delivering all of the software features, on time, with high quality, and on budget is a problem. Scott Ambler’s survey simply challenges the definition of success. (Projects that are challenged can also generate more than the anticipated benefits, but nothing is certain until your product is in use.)

Standish Chairman Jim Johnson was quoted about the improvement in success rates saying, "The primary reason is the projects have gotten a lot smaller. Doing projects with iterative processing as opposed to the waterfall method, which called for all project requirements to be defined up front, is a major step forward." (This comment was made prior to 2010, and I don’t have anything definitive about the dip in success rates. Does anyone have any insight on this?)

Focusing on the industry averages of Agile project success rates, the success rates are not perfect, but they are very good. The 2009 State of Agile Development Survey sponsored by VersionOne, includes data from 2,570 participants from 88 countries and reports that 23 percent of the respondents indicated that they had not experienced a failed Agile project. This is an improvement from 17.4 percent who reported no Agile project failures in VersionOne’s 2008 report, and in contrast to the Chaos Report’s downturn in overall software project successes.

I also took a look at the figures from Scott Ambler’s survey. Success in Scott’s survey was defined by the respondent, and not subject to a stricter definition such as the “on time, on budget, with quality and with the expected features” criteria used by the Chaos report. I’m fine with this, since a great many are arguing that the Chaos criterion is flawed.

I’ll stick to the Agile versus Traditional (plan-driven) comparison. In keeping with the spirit of “what’s working,” Scott’s survey included examining success rates by location – the closer the team, the greater the odds of success – summarized as follows:

Success Rates Agile Traditional Co-located 79% 73% Near located 73% 69% Far located 55% 55%
Co-located means that everyone, including primary stakeholders, are in the same room.

Near located means that everyone is in the same geographic area, and could be on different floors, in different buildings, or working at home, but could potentially get together each day for a group meeting if need be.

Far located means that people would need to plane to attend a physical meeting of the entire team.

Overall, these studies indicate that nothing is perfect, but Agile improves your odds of success, and greater project success contributes to the overall return of your organization.

Conclusion
Summarizing what’s working in general for Agile software development projects, I’d have to say that the findings support that smaller, co-located projects that deliver incrementally allow a greater and earlier return on investment, and have a slightly greater chance of success than larger projects and/or distributed project teams. Finally, practices such as pair programming and test-driven development should be utilized to achieve productivity gains.

What is your take?

Have you ever participated on a forum or e-mail exchange that became a great example of terrible communication? This happened to me recently with on Reddit when I submitted a link to my post, What is Better About Agile Development? (This post was a reprint of an article that I wrote for DevX.com titled, Top 10 Reasons to Use Agile Development.)

There are times when face-to-face communication is much more effective, where a conversation only minutes in length can save hours of electronic chatter. This Reddit dialog felt like one of those times.

It started with this comment about my article: “Methodology marketing - no 'softwareresults' - no evidence, just assertion.”

I responded with, “It's an opinion about how Agile development can provide a number of benefits – and improve the results of software development efforts. I structured this piece with reasoning to support the assertions; but no, I did not provide specific evidence. Does the reasoning fail to convince you? What evidence would work for you?”

Things went downhill (and around the hill) from there.

I’ll admit that my motivation was to explore whether the assertions and brief reasoning offered in the article held up with someone who clearly:

• Has experience in the software development field.
• Reads a great deal on the subject.
• Participates in forums and comments on blogs.

My attempts failed. My agenda was to explore my article on its own merit, while this individual wanted me to supply proof of my assertions. This individual eventually summed up my article with this: It's not even wrong.

I believe this to be a harsh judgment, and our dialog failed to convince me that anything that I said in the article was in fact wrong. This individual demanded evidence from me, but he didn’t present any evidence that my assertions were wrong based on research that he has clearly performed, either.

I do agree (in part) with his answer to this question that I asked: “How should the industry be approaching software development, in your opinion?”

He answered, “With humility and measurement.

"The Principle of Dispassionate Methodology: Don’t get your method advice from a method enthusiast. The best advice comes from people who care more about your problem than about their solution."

I happen to care about software development and meeting the needs of the business, and I’ve become enthusiastic about Agile development because I believe that when it comes to working with the business to meet the needs of the business, the approach is excellent. And I certainly don’t prescribe using Agile methods if they don’t fit your situation.

This individual was right about one thing. My blog is about software results, and I could back up my assertions more thoroughly than I did, and I will do so in my next series of posts. In advance of this, let me ask you a few questions so that I can learn to write more compelling and informative articles.

Please keep in mind that writing articles places some constraints on you as a writer. The editor had already determined the topic in this case. He wanted a “Top Ten” article (lists and “Top Reason” articles tend to be popular). There is a word limit. I was given a range of 800 words to 1600 words, and I came closer to the upper limit in this case.

I simply lacked the space to delve into details of each assertion. I had just enough room to explain each one as succinctly as possible. My hope for the article was that it would get someone excited enough to consider to want to explore more, or perhaps provide a convenient reference for those who understood and supported Agile development, but needed to explain the benefits of Agile development to others.

My questions to you:

In your opinion, does my article capture the benefits and reasons for using Agile, based on your understanding of software development and your experiences? Why or why not?

What would you recommend that I change?

What other articles would you like to see written?

I’ve been a software development manager for several years, with plenty of time invested each year on the annual review process. And like everyone else, I’ve been on the receiving end of the annual review process throughout my career. Being on the receiving end is the very foundation of the entire book!

Samuel Culbert points out that the typical performance review is not an interaction between manager and employee, that there is only one opinion that counts: the boss’s. Furthermore, the entire conversation is set up for failure, because the employee believes he or she is negotiating pay and readiness for advancement while the boss is focused on missed opportunities, skill limitations, and relationships that could use enhancing.

Culbert continues, noting that everyone lacks something as measured by other people’s metrics. Yet despite this lack, they are accomplishing a lot in their lives anyway. And when it comes to managing knowledge workers in particular, so-called “objective” reviews aren’t. You can get excellent and poor reviews by different managers, even if you are doing the same job. (Have you ever had this experience? I have.)

The result of this – and Culbert does a great job of articulating the problems – is that the performance review does exactly the opposite of its intended purpose. It actually prevents workers from improving. Culbert calls it “a dehumanizing process that leaves workers demoralized, unwilling and unable to address weaknesses. It makes them hate coming to work, let alone inspire them to turn themselves into better employees.”

One example Culbert uses is from an engineer, who said, “My desire to please everyone, to receive a stamp of approval from everyone, is my weakest point.” But, according to his boss, the problem is that he wastes too much time accommodating requests from other departments that should be doing their own work. In this guy’s mind, he’s just trying to help others out. But in his boss’s mind, he’s insufficiently focused. Who’s right? They both are. Everybody is right and everyone loses. There is no column on the performance review for “tries to please everybody,” but there is one for “lacks focus.”

What should be done instead? Culbert recommends using a performance preview.

The goal is to have an ongoing dialog, not a monologue, to align the manager and employee to accomplishing a common goal. This makes good sense to me. As a manager, why judge someone after the fact, from “on high?” Where were you before the act?

Some key observations about the differences between a performance review and a performance preview:

• Performance reviews are one-sided-accountable and boss-dominated monologues. Performance previews are two-sided conversations, with both sides accountable. 
• Performance reviews mean that if the subordinate screws up, the subordinate suffers. Performance previews put both the subordinate’s and boss’s skin in the game.
• Performance reviews lead to bullshit. Performance previews lead to straight talk. 
• Performance reviews allow the big boss to go on autopilot. Performance previews force the big boss to become involved.
• Performance reviews create a competition between boss and subordinate. Performance previews create a team where both teammates inform and learn from each other. 
• Performance reviews focus on deviations from some ideal as weaknesses. Performance previews celebrate differences.
• Performance reviews are about comparing employees. Performance previews treat people as individuals. 

Another component of the performance process that I think is appropriate, yet a strong deviation from the standardized review process in most organizations is the notion that “one size doesn’t fit all.” If you read this book and you feel that changes should be made in your organization, be prepared that specific recommendations aren’t available. It will get you thinking about the need for change and provide some solid guidance on what change should look like, but there won’t be a new procedure for you to follow. As the book noted:

There is no magic formula, no silver bullet. Ordering people to use a procedure in lockstep fashion would be the kiss of death because you would be replacing an antiquated formula with a new mandated formula. If people don’t have to think about the need that generates their activity, mindless stuff happens. Each management unit and each manager must come up with a format that they own.

The book is well-written and really makes an excellent case for getting rid of the performance review! (Of course, this is easier said than done.)

My recent posts have been made from a backlog that I established, but I'm beginning to run out. This is a temporary condition, one due to my father spending the last six weeks in the hospital, ultimately passing away. As a result, I am taking a short break until I can get back to writing more content. I have other material in various stages of preparedness, and I expect to continue in a week or so.

I recently participated in a discussion on LinkedIn, where someone asked, “Is Agile a methodology or a set of guidelines?” This provoked a number of responses, including:

“Agile is a framework rather than a methodology.”

“It is a set of principles rather than a methodology.”

“Agile is definitely just a framework.”

“Agile is not any single one thing.”

“Agile is not a methodology or a framework.”

“Agile IS 4 values and 12 principles, with frameworks that adhere to those Agile Values and Principles. “

So, just what is the proper definition of Agile development? Is it a framework, methodology, or something else? Is it nothing definitive at all?

Let’s start by examining what it means to be agile. Wikipedia defines business agility as: “the ability of a business to adapt rapidly and cost efficiently in response to changes in the business environment.” Wikipedia expands on this, stating, “Agility is a concept that incorporates the ideas of flexibility, balance, adaptability, and coordination under one umbrella. In a business context, agility typically refers to the ability of an organization to rapidly adapt to market and environmental changes in productive and cost-effective ways. The agile enterprise is an extension of this concept, referring to an organization that utilizes key principles of complex adaptive systems and complexity science to achieve success.”

Being agile in a business context equates to one of the primary benefits of Agile software development: that it strives to be responsive and adaptive to the business. While the purpose of Agile development may be well-understood; people get wrapped around the axel when they attempt to succinctly define Agile development, and disagreements surface when terms like “framework,” “methodology,” “guideline,” and “principles” become a part of the definition.

Wikipedia defines Agile software development as, “A group of software development methodologies that are based on similar principles.”

This is accurate, but it doesn’t quite work for me. I’ll come back to this definition in a moment. Let’s examine whether we should be using words like framework, methodology, guideline, and principles in the definition of Agile development. The formal definitions of each:

Framework – (Wikipedia was light on this definition, so I used Wiktionary.org) A framework is a basic conceptual structure used to solve or address complex issues.

Methodology – (Wikipedia) The study of methods used in a field.

Principle – (Wikipedia) A law or rule that has to be, or usually is to be followed, or can be desirably followed, or is an inevitable consequence of something, such as the laws of nature or the way that a device is constructed.

Guideline – (Wikipedia) Any document that aims to streamline particular processes according to a set routine. By definition, following a guideline is never mandatory (protocol would be a better term for a mandatory procedure).

Confused yet? I’ll keep going. Other definitions of Agile Software Development tend to be descriptive, like Scott Ambler’s: “Disciplined agile software development is an iterative and incremental (evolutionary) approach to software development which is performed in a highly collaborative manner by self-organizing teams within an effective governance framework with “just enough” ceremony that produces high quality software in a cost effective and timely manner which meets the changing needs of its stakeholders.”

Describing software development is fine. However, my objective is to provide a succinct definition, separating the what from the how, and to make a determination about whether Agile development is a framework, guideline, or methodology. In writing this blog, I’ve spent some time thinking about the terminology and my LinkedIn response, which in part was:

“Agile SOFTWARE DEVELOPMENT refers to a group of software development methodologies, whereas a specific implementation such as Scrum or XP IS a reference to a specific framework, methodology, or guideline (all of which I take to describe a process or approach to Agile development). I lean towards referring to something like Scrum as a methodology versus anything else. A guideline strikes me as too weak, and a framework invokes an image of static building-blocks rather than a process flow.

“My other two cents is that Agile development and the Agile Manifesto should not be confused, terminology-wise. But what the Agile Manifesto brings to the table is worth noting. The Agile Manifesto is an instrument to succinctly state the mission of Agile development. The values and principles of Agile development should be embodied within a specific Agile methodology, since those are the key tenets of Agile development. The values and principles reflect the spirit of Agile, and the implementation is the methodology.”

I was wrong!

Well, in part. I still prefer to stay away from defining Agile development as a framework or a guideline because Agile development does come in various flavors, including Scrum, XP, Feature Driven Development, etc. – and each have their own specific approach.

I also stand by my take on the Agile Manifesto. Agile development is more than a set of principles, although thanks to the Agile Manifesto we have a documented set of values and principles for Agile development. Ultimately, the definition of Agile development must be a general term that allows for specific implementations.

The use of the term methodology fits, if appropriately used. And this is where I was wrong and where I feel Wikipedia’s definition is slightly off. The term methodology has been used – inappropriately – as a substitute for method. Here’s a quick take from Wikipedia on the use of the term methodology:

Etymologically, methodology refers to the study of methods. Thus the use of methodology as a synonym for methods (or other simple terms such as means, technique, or procedure) is proscribed as both inaccurate and pretentious.

Agile software development is thus the study of methods – a methodology, whereas a specific implementation such as Scrum is a method. Wikipedia therefore should not refer to Agile software development as a “group of software development methodologies,” but as a “group of software development methods.”

What is my definition, using one sentence?

Agile development is a methodology that refers to a group of adaptive, responsive software development methods that value delivering working software in the shortest period of time in a sustainable manner.

Managers are more important to an employee’s decision to remain with a company than any other factor. The book, First, Break all the Rules, takes the position that “…if your relationship with your manager is fractured, then no amount of in-chair massaging or company-sponsored dog-walking will persuade you to stay and perform. It is better to work for a great manager in an old-fashioned company than a terrible manager in a company offering an enlightened, employee-focused culture.“

Since being a good manager is this important, here's my personal list of rules for better management.

1. Know your staff, and align their strengths and preferences with the needs of the organization. This means getting to know each person as an individual, seeking to understand their strengths, weaknesses, preferences, and career aspirations. If you can leverage peoples’ strengths and assign them to projects that are in line with their stated preferences, they will be motivated to perform, and more likely to be successful in the process. This will also result in less directing and cajoling on your part – a real win/win.


2. Understand what good performance looks like. As a manager, you don’t have to be the best performer, but you do need to know what stand-out performance looks like in order to judge it accurately. To do this, look to your best performers; you cannot understand what excellence looks like by studying failure.


3. Provide clarity and context relative to expectations. Make sure that you have you clearly communicated your performance expectations in terms of what needs to be done and how to go about it. This should be in the form of a regular, two-way dialog and not a "it's my way or the highway" mandate. As a part of this dialog, make sure that the expected results are understood in context of the organizational goals and values.


4. Stretch people, but do so carefully. To get high standards of performance you need to set tough, yet attainable goals.


5. Pay attention. I’ve seen articles and books dating back decades that discuss the fact that people don’t get the time and attention from their bosses that they feel they need. Some managers deliberately distance themselves from their staff, not wanting to be "too familiar" or too close because they feel it will undermine their authority. This doesn't work, as you cannot build personal influence and strong relationships by distancing yourself.
   
   Another part of paying attention to people is finding ways to provide praise and recognition for things that people are doing right on a regular basis. People need to know that you are paying attention to what they do, and that they can count on frequently hearing from you. Finally, giving praise and recognition are not only great motivators, but it makes any corrective conversation that needs to be conducted a whole lot easier.


6. Provide opportunities to learn and grow. Knowledge workers need to expand their horizons to be of continuing value; putting knowledge in the knowledge bank allows you to make those continual withdrawals, paying dividends in the process.


7. Identify and remove impediments. People sometimes need a little assistance and guidance to make them more productive. A few key questions to ask that can surface typical bottlenecks:
   
   
   • Does your staff have the proper materials and equipment required to do their work?
   • Are there things that you as a manager or another part of the “system” are doing to impede progress?
   • Is too much being demanded of too few?
   
   
8. Take the long view. Yes, we all have urgent tasks that creep into our day, but keep the longer-term objectives in mind when dealing with those day-to-day pressures. Continual firefighting and unchecked multi-tasking keeps people very active and busy, but it diverts you and your organization from achieving meaningful results. Ultimately, this lack of meaningful accomplishment will be a significant de-motivator.


9. Provide confidence and optimism. This is particularly important when the pressure is on. People and teams that don’t allow themselves to get rattled under pressure can make all the difference in terms of successful delivery and failure. Under times of stress, managers can shore up everyone by projecting confidence.


10. Assess employees on specifics, not on generalities. Give people something concrete to work with. “Rudeness” is general, whereas specifying the improper behavior of interrupting a co-worker when she was making a suggestion is specific. “Not a team player” is general, where pointing out that not volunteering on several occasions to take on extra work is specific. Having a “good attitude” is general, whereas smiling at people and never complaining during crunch time is specific.

Did I miss anything that you view as important?

This post is a reprint of the article Top 10 Reasons to Use Agile Development that I wrote for DevX.com, published on April 27, 2010 and used here with permission. Given the comments generated from my last post on this site and on Reddit, I thought it would be appropriate to provide my thoughts on the benefits of Agile development. And I'm very interested to learn your thoughts on the subject!

You’ve heard the buzz: Agile development is all about solving the problems associated with traditional software development. The Agile Manifesto states, “We are uncovering better ways of developing software by doing it and helping others do it.” But just what is better about Agile development?

As a software development manager, I’ve had to consider this very question as we’ve progressed from testing the Agile waters to a full-blown implementation of Agile across our organization in the past few years. My thinking and this list are the result of my personal experience with Agile, numerous conversations, training, and plenty of reading on the subject of Agile development written by other experienced, very capable practitioners.

1)      Superior ROI.
With many software projects, the business is forced to wait for the completion of the entire project before it can begin deriving a benefit from that investment. Given the challenges in meeting software schedules, those on the business side of the house not only have to wait for delivery of the software, they have the potential to be disappointed in a couple of ways:

a)      Features are trimmed from the final delivery. b)      Quality trade-offs are made.

(There are other ways to be disappointed, and many are addressed in this list.)

Agile emphasizes delivering early and often, enabling the business to begin generating a return on its investment much earlier. Agile does ask for discipline and participation from the business as part of the deal, such as rigorously prioritizing the features and being available to answer questions during the development cycle. In return, the business gets its highest-valued features delivered early, and delivered with quality.

2)      Business agility is embraced. In order to capitalize on opportunities, the capability for a business to adapt and respond to change is critical. Software development practices shouldn’t run counter to business needs by forcing the business to choose and adhere to a set of pre-determined features that will be delivered months, if not years, into the future.

Agile development welcomes and adapts to change. Because software is delivered in short iterations (measured in a few weeks) from a prioritized backlog of features, Agile development projects are able to easily adapt in accordance with changing business conditions.

3)      Agile development reduces risk. There are a number of risks (a.k.a., challenges) with every software project. Schedules, budgets, scope creep, and one of the most demoralizing for everyone involved – particularly if you’ve exceeded your planned schedule and budget – delivery of software where the users “got what they asked for, but it isn’t what they wanted” because the requirements were not understood in the first place. (This sometimes masquerades as “changes in requirements.”)

Agile development seeks to avoid these issues with frequent delivery of working software that allows the business users the opportunity to provide feedback based on frequent inspection. This permits the team to make immediate corrections if there was a misunderstanding.

The frequent delivery of working software also keeps schedules and budgets in check. There is complete transparency and certainty about the progress of the team because the features delivered slice vertically through the architectural layers. This eliminates the late-stage integration and quality issues suffered with software projects that use different delivery mechanisms.

Finally, the business has the option of discontinuing further investment and retaining what has been delivered at any time, instead of being forced to make a full (or greater) investment in order to build out the software to a state where it is suitable for use.

4)      Agile development increases productivity. Producing software that meets the needs of the business requires the involvement of knowledge workers from a variety of disciplines – business and technical – to work together. Agile development focuses the team’s attention like a laser on delivering the highest-priority, highest-valued features in short increments of time using the most productive mechanisms to accomplish the work.

As part of this delivery, Agile development goes beyond using directed teams that are in reality nothing more than a collection of individuals working on assigned tasks. Agile teams embrace collaboration in the truest sense of the word; there are shared goals, shared knowledge, shared learning, shared progress, and a shared responsibility by the team to meet its commitments.

Another productivity increase comes from teams operating autonomously, where the teams make informed decisions about their day-to-day work without the need for constant managerial direction. Not only does this expedite certain decisions by keeping them localized, employees who have more control over their day-to-day work are more energized and engaged about their work.

When an autonomous, highly collaborative, multi-disciplinary team is firing on all cylinders, the productivity gains can be extraordinary.

5)      Agile development creates a sustainable development environment. Software projects rarely meet their initial schedule. Overtime with the mandate to “work harder” is frequently used as it becomes apparent that a project will be running longer than anticipated. Often times, frustrated managers feel justified in holding those who provided the estimates at fault for not meeting their commitments – in spite of the fact that an estimate is supposed to be an approximation and not a precise figure.

There are a number of problems with the regular use of overtime, including more mistakes being made by tired employees, risk of costly turnover, and the simple fact that a constant overtime model is not sustainable in the long run.

Agile development sizes User Stories in points and tracks team velocity – the User Story points that are completed in each 2-4 week iteration. Over a period of time, it becomes crystal clear how much work can be realistically accomplished by a team on a sustainable basis. The goal then becomes one of working smarter to improve, not harder.

6)      Agile development enables emergent innovation. Big innovations are easy to recognize, but hard to come by; they typically materialize from work outside of day-to-day projects. Because Agile creates a sustainable development environment, a greater opportunity exists for innovation to emerge from the employees. Teams that are working constant overtime to meet schedules simply lack the time or inclination to think about anything else other than the difficult schedule in front of them. In sustainable development environments, people have the time to think more about the business and explore – creating the potential for innovation that did not exist previously.

Even on “routine” tasks, the collaborative nature of Agile development creates the opportunity for smaller innovations to occur. Requirements are discussed as User Stories, involving what the business needs along with the benefit that it hopes to realize. The important aspect of this is that the requirements aren’t considered to be cast in concrete; there is a discussion about what the business is hoping to achieve.

The discussion yields important insight and understanding for the entire team. The worst-case scenario is that everyone will be on the same page. At other times, the dialog between business experts and the technical experts can yield unexpected results, like turning complex, difficult features into elegant, differentiating features.

7)      Agile development builds trust and relationships. Through early and often delivery of working software to the business – even if this is nothing more than demonstrating the features – the business will grow to trust the development team. In addition, the continual dialog and the ability for the business to adjust and adapt the software gradually changes the “us versus them” dynamic into a “we.”

The same will happen for the members of the Agile development team. Instead of everyone divided by functional roles, Agile teams make the most effective use of the team members as dictated by the needs of the team to meet its commitments. The shared goal becomes more important than each individual working strictly within his or her area of expertise, with defined policies and procedures for handing off work between functional roles. This breaks down barriers between functional disciplines, enabling the team to reach higher levels of productivity.

8)      Agile development expects continuous improvement. Agile development expects its practitioners to be continually learning and adapting, and is something that is enabled in part through sustainable development. Sustainable development provides the time and energy for the development team to expand their working knowledge of their profession through self-learning.

In addition, Agile teams conduct regular retrospectives at the end of each iteration to review what is working well and what can be improved.  Team members are expected to assess their teamwork and their use of (or lack thereof) technical practices and to make adjustments in the upcoming iteration to improve.

9)      Agile development is motivating and engaging. Agile recognizes that knowledge workers have the greatest understanding about their own work, and that they are the ones best qualified to plan and organize how they will accomplish that work. Agile development utilizes an autonomous, self-directed work environment that is a powerful motivator. This control over their workday, plus operating in a sustainable mode and the feeling of accomplishment that is a by-product of continuous delivery of working software, all combine to provide a solid motivational boost to each and every person on an Agile team.

10)   Agile addresses the realities of software development and the needs of the business. The challenges that every software project faces stem from the fact that we’re not producing pre-defined widgets; there is uniqueness involved with every software project. Agile development’s entire approach addresses the major problems encountered with software projects and managing talented knowledge workers while being aligned with, and responsive to, the business.

I’ll close where I opened. There is a reason that the Agile Manifesto states, “We are uncovering better ways of developing software by doing it and helping others do it.” Agile is a better way.

I find it fascinating when I hear someone state that “Scrum doesn’t work” or that it isn’t all that it is advertised to be, particularly in organizations that have embraced and adopted Agile/Scrum development. It’s always worth exploring why that person feels the way he or she does, particularly if Scrum isn’t working for a software development project.

If a lightweight process like Scrum isn’t working, you’re seeing the fragile side of Agile. It’s all too easy to read about Agile development and listen to people like me – I’m an optimist by nature – champion the wonders of Agile development and how it will improve your development universe. While Agile sounds simple on paper, a proper implementation requires a deep understanding, discipline, and organizational support. And even if you get it right initially, it's easy to go astray.

I embrace Agile/Scrum development because I am firmly convinced that it addresses many of the problems that plague software development along with providing working professionals with the autonomy that they should have in the first place. If something isn’t working, most likely it isn’t the process that is at fault. I’d wager that there is an implementation or execution problem.

We’ve implemented Scrum at my company, and we’ve been fairly successful with the basics to date. Unfortunately, I keep hearing and reading about other organizations that are failing on the basics, hindering their successful adoption of Agile/Scrum. This is the classic ScrumBut problem of, “We’re doing Scrum, but…”

• We don’t have a product owner.
• We don’t have a product backlog.
• We don’t bother with daily standup meetings.
• We’re not doing ... some fundamental activity that is and should be a part of the Scrum process.

Many organizations struggle with implementing the basics. Jeff Sutherland has spoken frequently about the Nokia Test and how upwards to 80% of teams that claim to be doing Scrum don’t perform the basics (some don’t even know who the product owner is, let alone have a prioritized backlog).

I’ve witnessed the benefits of Agile/Scrum, seeing teams improve productivity, reduce overtime, and generally enjoy their day. But we have experienced problems as well.

A common problem is that teams will implement the mechanics of Scrum in the context of old habits. In the software world, functional specialties become the focal point: analysts create the requirements, developers develop the software, and QA tests the software once development is “done.”

Functional hand-offs turn Agile projects into a series of “mini-waterfall” efforts – bypassing the collective, collaborative aspects that Agile development brings to the table to boost productivity. The risk with this approach is that the User Stories may be in various stages of completion at the end of the sprint, but aren’t “done done.”

Even when teams are collaborating well, they sometimes take on more work than they should. This problem surfaces in different ways.

• Teams fail to track their velocity – skimping on an important aspect of Scrum – and then continually misjudge their capacity to take on work and end up committing to more User Stories in a sprint than they should.
• Teams start work on too many stories at the beginning of the sprint instead of focusing on the high-priority stories, possibly feeling external pressure to get a set amount of work done in a specific time frame. This again leads to ending with User Stories in various stages of completeness, none of which qualify as finished.

The point about external pressures is also important relative to the successful adoption of Agile development. The surrounding organization can have an impact. Martin Proulx provided some great examples in a recent post about Agile mis-adoptions he's encountered as a consultant that are as much fun as they are scary. One of my favorites:

Client: We have identified a list of issues that we need help with. Here’s the list. Can you help us?
Martin: Possibly. Let me look at your list. Who came up with the items on this list?
Client: Me and my direct reports.
Martin: Has the team been involved in putting this list of issues together?
Client: Absolutely not. We asked them to put together a list of issues they were facing and most of the items were related to lack of trust, micro-management, and bad communication so we threw out their list and put this one together for them…

Or this one from an overly-enthusiastic manager:

Client: This Agile thing is great! I’m going to impress the management team with our success.
Martin: How so?
Client: The development team asked me if they could use Agile for their next project and from what I read, Agile can help them improve their performance and reduce the time to market.
Martin: Yes, if it’s done right you may get those benefits.
Client: Wonderful! After I gave them the go ahead to start immediately, I told them I now expected to project to be delivered in 9 months (instead of 18 months) and cut their budget by half…

When it comes to problems with Agile development, people tend to blame the process when in reality the problem(s) are in their implementation and/or support of the process. While it takes more than you realize to fully implement Agile development correctly, the potential that it creates for greater productivity along with aligning development with the business makes in investment worthwhile. It forces you and your organization to contend with the very issues that are preventing you from improving today.

by Jack Bergstrand. Copyright © Brand Velocity, Inc.

As a manager seeking to improve my game in managing software developers, I’ve taken the stance that software development is knowledge work. I therefore keep an eye out for useful information about managing knowledge workers, not just software developers. The book, Reinvent Your Enterprise Through Better Knowledge Work appealed to me on that basis.

Jack Bergstrand makes some solid observations about the need for productively managing knowledge work. One being that the half-life of knowledge continues to get shorter. This places demands on knowledge workers to learn faster, interact better, and produce better and more accelerated results. (This struck a chord with me, since we’re always looking to produce “better software, faster,” and the rate of technological change has been significant during my career.)

And while this wasn’t a software development book, another observation was spot-on: Knowledge work is how individuals and groups use ideas, expertise, information, and relationships to get things done. It includes tasks such as brainstorming, analysis, project management, and personal coaching.

These activities are exactly what we’re doing with our Agile/Scrum development process. We want individuals to collaborate and share ideas and expertise.

Bergstrand notes that knowledge work actually needs to be managed better than manual work because there are so many ways for it to go off track and become unproductive:

• Too many meetings that produce too few decisions and actions
• Competing internal priorities with no mechanism for resolution
• Studies that are completed and put on the shelf
• Projects that get started but are never finished
• Projects that get started but are not finished on time
• Projects that never get started but get talked about every year
• High executive turnover that causes frequent changes in direction

A few thoughts/recommendations Bergstrand puts forth:

A system to manage knowledge work requires both a shared framework (shared mental model) and an explicit process. The framework is needed to get everyone on the same page. The process helps people manage their knowledge work more productively and sustainably. The model needs to simplify while also being robust enough so that knowledge work tasks can productively self-organize and the architecture in a variety of situations and under various conditions.

It is usually productive to write off underperforming products, operations, and orientations at the same time an organization is moving forward with new things. This can help establish new standards of performance.

Organizational hierarchies produce known problems, but it is not productive to be anti-hierarchy. Working without a clear organizational structure is the most unproductive situation of all.

Overall, this book was well-written, and provided me with an excellent, generic perspective on managing knowledge work, but less so on managing knowledge workers themselves.

High-performance business teamwork demands a great deal from individuals. If you want a high-performance team, it needs to be staffed with competent people who have clarity in terms of direction, understands what needs to be done, and are capable of performing that work.

As a team member, what are the characteristics required of you to be perceived as a valued, contributing and positive force on that team? Be…

1. Engaged. Demonstrate a willingness and desire to advance the team’s goals by proactively seeking to add value to the work that the team is performing. Don’t wait for someone to assign tasks to you, be a self-starter. Seek to understand the objectives of the team and volunteer to take on tasks that are within your ability to perform; and don’t shy away from tasks that may stretch you a bit.


2. Action-oriented. Have a bias for accomplishing work that provides value, avoiding the "analysis paralysis" trap. This does not mean taking hasty action, such as bypassing research and planning. Both can add value, but seek to understand what you need and how much is really required. There is no such thing as perfect information or a perfect plan, and you should move forward when you have sufficient information; seek advice from those with experience to help define “sufficient” for your situation.


3. Committed. Make a commitment to delivering value by a certain date. Through engagement, action and commitment you will become a contributing team member.  This includes making sure that the team as a whole is pulling together to meet its overall commitments. If you are confused about what constitutes value to your team, by all means ask the question!


4. Accountable. Hold yourself personally accountable for meeting your commitments and hold others to meeting their commitments. If there are problems, inquire about what is causing a problem – but keep the discussion depersonalized. Accountability is expressed as “I will…” and not, “I’ll try…”


5. Collaborative. You need to be someone who is both a good listener and willing to speak your mind. Engaged team members understand the work, form opinions and voice their opinions. It is not in the best interest of the team to have a select few (or one) forming opinions and driving direction. Keep in mind that when there are multiple opinions in play, others may have different perspectives based on their experiences and understanding. There very well may be aspects all of the opinions that should be considered.


6. Adaptable. The business world is a dynamic, fluid world, and the needs and dynamics of one team will likely be different than other teams. Be prepared for changes in priorities and be coachable – you may need to adapt your personal style for the good of the team.


7. Supportive. Teams are comprised of individuals with complementary skills, and there will be times when individuals will be operating outside of their comfort zone. Support people by providing assistance in the form of guidance and coaching, but do not cross the line into carrying someone else’s load (this disrupts accountability). Be supportive of the team in general as well, including its goals and methods. Bad-mouthing the team to others will come back to haunt you, and demonstrates a lack of commitment to the team. If something is making you uncomfortable, talk to the team about it, and in necessary, find a way to remove yourself from the team.


8. Transparent. Be open about any issues and the progress of your work, particularly when you are working on a task independently.  Your teammates cannot support you if they have no visibility into your work, and you will deny yourself the opportunity for valuable input from others.


9. Honest. Be truthful and sincere, and by all means keep your word!


10. Trusting. To be a good teammate, you must be able to trust those on your team to “play their positions.” If all team members have the above behaviors, there is no reason not to trust them. If there are problems, talk about them in an open, honest, no-personal way. Explore why there is a breakdown and what can be done about it.

You might notice a couple of typical “behaviors” missing from this list: Leadership and continuous learning.

Leadership is a quality that is spread throughout the behaviors; leadership emerges to the degree in which someone is committed, collaborative, etc. Leadership can also take other forms, such as possessing and sharing deep domain knowledge or (more than likely) a combination of both knowledge and behavior.

Continuous learning/improvement is a trait that can and should be embraced by both individuals and teams. Individuals should have something to contribute to a team, and continuing to learning more about your profession will increase the value of your contributions to a team. Personal and professional growth exhibits engagement in your chosen profession. In addition, teams should never be satisfied with their status quo, but should always be asking, “How can we operate better?”

Have you ever developed a passion for an idea as a result of considerable time and effort expended in research and thought on your part? Did your research and thinking lead you to a firm, unshakeable conclusion? Does your routine research and understanding of your profession provide you with solid opinions about how things should be done? Does your hard-earned insight occasionally blind you to alternative ways of thinking?

Guilty as charged.

You might work very hard at keeping current and forming opinions on various topics, but others will not have your perspective. And they won’t enjoy dealing with someone that they perceive to be so opinionated that alternatives aren’t at least explored and considered.

It is always good to keep an open mind and explore options, particlularly during the decision-making process. There is a common problem with us humans, and that is to focus too heavily on one piece of information during our decision-making process. This is known as anchoring.

If you place too much importance on one aspect, the anchor becomes “set,” and you will find it difficult to mentally shift away from your perception. This becomes more difficult when emotions are engaged – you feel excited and energized because you’ve had that inspirational flash of insight, and before you realize it, you’ve dropped your anchor and your ship isn’t going to be moved, at least not easily.

There is a difficult balance to maintain here. Productive workplaces are filled with people who have strong opinions. You certainly don’t want to have a bunch of people walking around without opinions – talk about a real lack of engagement! The trick is to adopt Paul Saffo’s mantra of having “strong opinions, weakly held.”

Paul’s advice: “Allow your intuition to guide you to a conclusion, no matter how imperfect – this is the ‘strong opinion’ part. Then – and this is the ‘weakly held’ part – prove yourself wrong. Engage in creative doubt.”

Strong opinions are healthy, and we should all develop the best possible reasoning and evidence to support that reasoning. Just don’t too attached to what you believe, because that attachment will undermine your ability to acknowledge additional evidence that is in conflict with your opinion.