In This Issue
- Is Agile Really that Successful?
- Hot Links
Is Agile Really that Successful?
My December 2007 magazine column summarized Dr. Dobb's 2007 Project Success Rate survey which showed that Agile software development teams have a much higher success rate than traditional software development teams. Since the article came out I've received a fair bit of feedback about it, some congratulated us on exploring how IT organizations really define success (the real goal of the survey), some tried to explain the differences between the two success rates, and some claimed that the results merely reflect the biases of our readership towards Agile approaches. Although a minor part of the survey, the differences between the apparent success rates of Agile and traditional approaches clearly garnered the most fervor, particularly within the traditional community, so I thought I'd explore this topic in greater detail in this newsletter.
The questions around potential bias are clearly the most serious because they're very easy for people to make, even when they're not true, and they're very hard to disprove.
- First, no matter what you do all surveys will have some bias because they only capture the opinions of people willing to fill out surveys.
- Second, the survey would in fact reflect the bias of the Dr. Dobb's readership. But, as previous surveys have shown, the Dr. Dobb's readership is varied, ranging from very agile to very traditional, from junior to very senior people, from programmers to IT management, and even includes some business stakeholders. One of the reasons why I use the Dr. Dobb's mailing list for my survey efforts, I have access to several other sources, is because I believe that Dr. Dobb's does in fact have one of the best mailing lists in the IT industry.
- Third, I received design help from several experts and I think we managed to avoid some of the common problems which plague many surveys. Furthermore, this isn't the first survey that I've done and I've taken training on survey design in the past.
- Fourth, if you don't believe me then I invite you to perform your own analysis. The source data, with the exception of identifying information, and the original questions as they were posed is available at www.ambysoft.com/surveys/. Whatever bias there is in this survey it clearly isn't being hidden from you.
But this still begs the question of whether the results were biased, so I decided to analyze the data even further. I originally reported a success rate of 71.5 percent for Agile projects and 62.8 percent for traditional projects, a difference of 8.7 percent. 586 people responded to the survey, and the Agile success rate was calculated from the responses of 351 people (59.9 percent of respondents) and the traditional rate from 500 responses (85.3 percent of respondents). So, if there's any bias in respondents, it appears to be towards having traditional experience over Agile experience.
I also decided to compare the results of people who just had Agile experience and who just had traditional experience, and some interesting differences emerged. The 15 respondents with Agile-only experience reported a success rate of 84.3 percent. Although the Agile-only response rate is too small to be statistically valid, it is still interesting to speculate as to why there is such a significant difference. Perhaps these people are in small organizations doing straightforward projects. Perhaps they might truly be in an environment where they only take an Agile approach, implying that the project teams don't have to contend with the rest of the organization's traditional structure and culture. I don't know, but it's something that's worth looking into.
The 164 respondents with traditional-only experience reported a success rate of 66.5 percent, a step up from 62.8 percent although still not the 71.5 percent success rate of Agile projects. Once again I don't know why this difference exists, but a plausible explanation is that these people work in organizations which have chosen to streamline their development process around traditional strategies. Perhaps they're taking a CMMI-based approach which follows the philosophy that having a repeatable process is critical to your success. An interesting observation about the CMMI community is that to my knowledge they've never published comparisons of organizations that were effective at system development but had not taken a CMMI approach with CMMI organizations. All of the studies that I've seen compare the effectiveness of various organizations at different CMMI levels, but never with non-CMMI organizations. This isn't to say that you can't take an Agile approach to CMMI, but it is rare in practice.
Arguably the least bias would be seen with the results from people who have experience with both Agile and traditional development, which in this case was 336 respondents. This group reported success rates of 70.9 percent for Agile and 61.1 percent for traditional. So, based on this data, there appears to be a risk premium of 9.8 percent associated with traditional development. Also, notice that the success rates are lower than the average in both cases, potentially indicating that there's a risk premium for supporting both paradigms simultaneously, one of the costs of moving from traditional to more agile approaches.
One explanation for the higher success rate is that Agile teams are often made up of more effective people. The teams composed of your best and brightest will usually outshine the teams that aren't. One advantage that agilists have over traditionalists is they tend to be generalizing specialists, people with one or more specialties such as programming or modeling, a general knowledge of software development and the domain that they're working in, and a desire to collaborate with others and learn new skills as a result. Because they are more skilled than simple specialists they're in a better position to apply the right technique to the appropriate level for the situation at hand. Specialists, on the other hand, are limited to their specialty and as a result perform that specialty whether they really need to or not. Writing detailed formal use cases, or collecting detailed meta data to support your logical data model, may seem like great ideas to the specialists promoting them, but generalizing specialists would redirect that effort to other activities which provide greater value to your project.
Per Kroll, Manager of Methods at IBM Rational, had an interesting insight into the higher success rate. He believes that although Agile captures many practices that drives higher success rates that agile projects are often are smaller, and that as project size grows the success rate goes down. Agile teams are often smaller because they're more efficiently organized -- generalizing specialists require less bureaucratic trappings to organize themselves and as a result you require fewer people to get the job done. Kroll is sure that as we see more large agile projects we will see more troubled agile projects, a belief that was borne out by the Dr. Dobb's 2007 Agile Adoption Survey which showed that larger Agile teams did in fact have a lower success rate than smaller ones.
It's always good to be skeptical about the results of any research efforts, and in particular the types of surveys that I've been doing. At the same time, you also need to be open minded, particularly when there doesn't yet appear to be any better numbers available to us from other sources. I believe that many traditionalists are threatened by Agile approaches, often because they don't understand what agile is really all about or sometimes because they're comfortable with their current development paradigm and don't want to change. But with the mounting evidence that Agile approaches work better, I'm not sure that they're going to be able to hold out for much longer.
Hot Links
Defining Success summarizes the results of the 2007 Dr. Dobb's Project Success Survey.
Survey Says... Agile Has Crossed the Chasm summarizes the results of the 2007 Dr. Dobb's Agile Adoption Survey.
Agile on a Fixed Budget provides strategies for taking an agile approach to development even when the budget, schedule, and or scope is constrained.
Generalizing specialists are described in detail here.
The Agile Alliance homepage is the best starting point for anyone interested in learning more about agile software development.
The Agile Models Distilled page provides links to overviews of a wide variety of models.
The Principles of Agile Modeling v2 are described here.
The practices of Agile Modeling v2 are described here.
Check out the Agile Modeling mailing list here.
