In This Issue
- Questioning "Earned Value Management" on IT Projects
- Hot Links
Questioning "Earned Value Management" on IT Projects
Earned Value Management (EVM) is an objective technique used for measuring project progress. EVM has its origins with the U.S. Department of Defense (DoD) as far back as the 1960s, although the term EVM was only coined in the late 1980s. EVM has been in the Project Management Institute's A Guide to the Project Management Body of Knowledge (a.k.a. the PMBoK Guide) from the very first edition in 1987, making it a staple of project managers' intellectual toolkits for over two decades. EVM can be applied on any type of project, although I will limit the scope of this discussion to IT projects only. Some (if not all) of the issues that I identify below may be applicable to non-IT projects.
EVM is a four step process.
- First, you define the work to be done in the form of a detailed project Work Breakdown Structure (WBS) or equivalent.
- Second, you prepare cost estimates for each item in the WBS (called "planned value" or PV), encouraging your teams to perform "realistic" estimations as you'll be measured against these estimates later; for example, on a traditional project, the creation and acceptance of your requirements specification may be worth 15 points out of a total of 237 for your project (or 6.3 percent if you choose to work on a percentage basis).
- Third, you must define the earning rule for each work item, although most organizations choose a single earning rule for each project and sometimes for all projects to simplify and make consistent the EVM calculations; for example, you may choose to get credit only when the item is complete or can you take credit for half the work being done at the point where half of the work has been performed, strategies called the 0/100 rule and 50/50 rule respectively. Alternately, you may come up with another strategy on your own -- personally, I prefer the 0/100 approach as I'm a firm believer in the agile philosophy that "done means done."
- Fourth, you execute your plan and measure progress along the way.
Proponents of EVM claim that it is an objective approach to communicating progress to stakeholders, enabling you to declare that you're X percent done at any given point in a project. They also claim that it can be used to determine whether a project is on budget and on schedule, potentially providing an early warning of potential performance problems. As I argue in this newsletter, I don't believe that EVM is very good at these things for IT projects, particularly agile ones. However, I do admit that EVM may the best option for traditional teams due to the inherent opaqueness of the methods followed by those teams.
In theory, EVM sounds like a great thing. You're earning value after all, and who can argue against that? In practice, however, EVM is at best a euphemism for tracking budgeted costs expended and at worst it's simply a justification for bureaucracy. To see why this is true, let's examine the claims of EVM proponents.
The first claim is that EVM enables you to objectively measure the value being produced by project teams. Their inherent assumption is that these project teams are actually earning value for your organization, yet in practice we know that this couldn't possibly be true. To see what the problem is, let's work through an example. Let's say that your team is part way through a project, claiming that they've achieved 65 percent earned value (EV). Your team has followed a traditional approach to development, creating a detailed requirements document, then an architecture model, then a prototype, then a design model, and now they've started working on the actual software and are about three quarters of the way through. You claimed 14 percent EV once the requirements document was accepted, 20 percent EV at the point the architecture was accepted, 33 percent when the detailed design was signed off, and will eventually claim 85 percent EV once the programming effort ends. You've had several major code reviews and by passing them you've been allowed to make the claim that you've achieved 65 percent EV. Then, disaster strikes and you lose funding as the result of an economic downturn. Senior management asks your team to ship whatever you can, but several critical components haven't been coded yet and still need several months of development. And you would still need to perform system and user acceptance testing -- another few months worth of effort. It's clear to senior management that none of their investment can be realized quickly, so the project artifacts are "put on the shelf" in the hope that some day they'll be useful to a future project team. Of course, by then the business requirements and technology platforms will have changed. In short, even though your team had claimed 65 percent EV, no actual practical value had in fact been "earned".
The second claim is that EVM is an effective way to track whether your team is on budget and on schedule. This is, in fact, true, although it's based on the assumption that you had identified an effective plan in the beginning. We know from hard-earned experience that this is a very large assumption when it comes to IT projects. To create an accurate plan, you must have a good understanding of both the requirements your project is to fulfill and the technology that you're going to use to fulfill it. Although the technology issues generally prove to be straightforward, albeit with the occasional hiccup along the way, understanding the requirements early in a project proves to be difficult at best. The reality is that people are not very good at defining up front what they want, that the business environment will change, and that how people understand their requirements will change throughout an IT project. If we do decide to freeze the requirements early in a project, following what is known as a "big requirements up front (BRUF)" approach, the result is that, on average, a large percentage of the functionality delivered by the actual team is never used. The Standish Group has found that with a BRUF approach, an average of 45 percent of the total functionality delivered is never used, and that 19 percent is rarely used, implying that teams who take a BRUF approach produce 55 percent "earned value" (at best) regardless of what they actually claim.
So, if EVM isn't a practical option for IT projects, what should we do? First, as the agile community likes to point out, the only true measure of progress on a software development project is the delivery of working software. Agile teams choose to produce potentially-shippable software at the end of each iteration, providing concrete and visible feedback to their stakeholders as to their actual progress. My experience is that the majority of stakeholders prefer this sort of tangible evidence of progress instead of intangible numbers. Second, recognize that the majority of our stakeholders aren't really interested in whether we're on schedule or on budget. Heresy you say! No, just a willingness to question the assumptions of the project management community. In August 2007, Dr. Dobb's ran a survey exploring how people define IT project success, and we found that 80 percent of people preferred to focus on producing a good return on investment (ROI) than being under budget, and that 62 percent wanted teams to ship their systems when they're ready to be shipped rather than forcing adherence to a schedule. In other words, the priorities are to spend the money wisely and ship quality systems. (When given the choice, 87 percent of respondents indicated that shipping high quality software is more important than being on time and on budget). The implication is that instead of measuring progress against your plan, as EVM enables you to do, you should instead be focusing on ensuring ROI and product quality. The agile practices of implementing requirements in priority order and allowing requirements to evolve throughout the lifecycle ensure greater ROI, and agile practices such as test-driven development (TDD) and refactoring promote greater quality.
Tracking EVM is clearly better than doing nothing, but it's an incredibly fragile metric at best because it assumes that the project team is actually producing a potentially-shippable system at all times during the development lifecycle and that the original plan is reasonably accurate. The first assumption is true of agile teams but is questionable at best for traditional teams. The second assumption is incredibly naive regardless of your development paradigm. In short, EVM is highly questionable for IT project management. As IT professionals we can and should do better. If we need to use euphemisms such as "earned value" to sell our projects, then perhaps it's time to euthanize the methodologies of yesteryear and instead adopt agile strategies that actually produce visible value to our stakeholders.
I'd like to thank Kevin Aguanno, the Agile Center of Competency lead at IBM Canada Global services and editor of Managing Agile Projects for his valuable feedback regarding this newsletter.
Hot Links
The Wikipedia page on EVM is a good starting point for learning more about the subject.
My April 2007 newsletter, The Dire Consequences of a Fixed-Price Project, explores the impact of too much up-front planning on IT projects.
The article Agile Requirements Change Management shows how agile teams focus on producing greater ROI by implementing requirements in priority order, shows how to manage changing requirements effectively, and describes why requirements "change" throughout a project in the first place.
The book Managing Agile Projects encompasses a collection of articles about agile project management written by leading voices within the agile community.
My December 2007 column Defining Success overviews the results of Dr. Dobb's 2007 Project Success Survey.
The questions, a summary slide deck, and the original source data from Dr. Dobb's 2007 Project Success Survey are available for free download.
See the principles behind the Agile Manifesto, including the idea that the primary measure of progress is the delivery of working software.
See my Agile@Scale blog.
The Agile Alliance homepage is the best starting point for anyone interested in agile.
The Agile Models Distilled page provides links to overviews of a wide variety of models.
The Principles of Agile Modeling v2 are described here.
The Practices of Agile Modeling v2 are described here.
Check out the Agile Modeling mailing list.
