Defining Success

How do you define project success? Do you consider a project to be successful when it is on time, or is it more important that a system is delivered when it is ready? Is it important to be under budget or should you maximize return on investment (ROI)? In August 2007, I decided to explore how the IT industry defines project success via an online survey (see the sidebar "Survey Respondents" for a description). Although I would have liked more responses from business stakeholders and data professionals, I was still happy with the fact that I received almost 600 responses. As with any survey, the results included a few unsuspected surprises as well as a few important insights into what is actually happening out there in the IT industry.

For years, we've been told that delivering on time and on budget is an important measure of project success. On the surface, this sounds like a good idea, but as you'll soon see it's obvious that these issues aren't as important as they've been made out to be. It would be naïve to expect us to define an industry-accepted definition of project success because the fact is that individual project teams find themselves in unique situations, implying that their definition of success will differ from that of another team. However, it is reasonable to expect to define potential success criteria, and I think that this survey has explored several very important ones.

How We Define Success

By far, the most popular source of IT project success rate statistics is the Chaos Report by the Standish Group (www.standishgroup.com). The third version of this report, published in 2003, notes that 34 percent of IT projects are successful, 51 percent challenged (they are over schedule, and/or they are over time, and/or they are missing significant functionality), and 15 percent of projects are considered failures. These figures never really seemed right to me, even though I'm embarrassed to say that I have referred to them several times over the years, because the Chaos Report was always the generally accepted source. But, if organizations don't actually define project success in the same way that the Standish Group does, then it begs the question what the actual success rates are.

When formulating this survey, I spoke /e-mailed with a number of people as to how they defined project success. I also scoured the literature for various writings on the subject, and as a result of this work I identified five critical factors upon which IT projects are typically judged. These five factors are: schedule, money (resources), scope, quality, and staff. This list isn't complete (for example, a project is sometimes judged on its ability to prove the viability of a technology or technique), but I believe that it does cover the common success criteria.

When it comes to these five success criteria, the survey found:

• Schedule: 61.3 percent of respondents said that it is more important to deliver a system when it is ready to be shipped than to deliver it on time.
• Scope: 87.3 percent said that meeting the actual needs of stakeholders is more important than building the system to specification.
• Money: 79.6 percent said that providing the best return on investment (ROI) is more important than delivering a system under budget.
• Quality: 87.3 percent said that delivering high quality is more important than delivering on time and on budget.
• Staff: 75.8 percent said that having a healthy, both mentally and physically, workplace is more important than delivering on time and on budget.

In short, we appear to value delivering high-quality working software that meets the needs of our stakeholders in a cost effective manner while respecting the needs of the people involved with the project. This is more important to us than building something to specification on schedule and on budget, regardless of the toll it takes on the people involved. Another way to look at it is that people appear to be much more aligned with the philosophies of the Agile community than we are with the rhetoric that we often hear within traditional circles. And we certainly don't like death marches, but then again who really does other than carrion birds?

The survey also asked respondents to rank the five success criteria in order of priority. Overall, respondents indicated that quality was the most important issue, then scope, staff, time, and money respectively. Table 1 compares the rankings of various subsets of the respondents, showing that commercial/private, government, and IT management all indicated the same rankings as the overall group. Stakeholders, nonmanagement IT, and project management all had different rankings, although all categories that I looked at indicated that quality was the most important issue by far.