Our infrastructure is essentially developed. The easy problems have been solved. Designing systems today is difficult because there is no consensus on what the problems are, let alone how to resolve them."
The year is 1973 and the topic is public policy. In their landmark article, "Dilemmas in a General Theory of Planning" (Policy Sciences, Vol. 4, Elsevier Scientific Publishing Company Inc., Amsterdam, 1973), Horst Rittel and Melvin Webber observed the type of problems that can't be resolved with traditional analytical approaches—and dubbed them "Wicked Problems."
It seems that wicked problems aren't unique to public policy. In their wonderful book, Wicked Problems, Righteous Solutions (Prentice Hall, Yourdon Press, 1990), Peter DeGrace and Leslie Hulet Stahl noted that many of the systems problems facing software developers have all the characteristics of wicked problems. Judge for yourself.
Wicked Problems
According to Rittel and Webber, wicked problems have 10 characteristics:
- Wicked problems have no definitive formulation. Formulating the problem and the solution is essentially the same task. Each attempt at creating a solution changes your understanding of the problem.
- Wicked problems have no stopping rule. Since you can't define the problem, it's difficult to tell when it's resolved. The problem-solving process ends when resources are depleted, stakeholders lose interest or political realities change.
- Solutions to wicked problems are not true-or-false, but good-or-bad. Since there are no unambiguous criteria for deciding if the problem is resolved, getting all stakeholders to agree that a resolution is "good enough" can be a challenge.
- There is no immediate or ultimate test of a solution to a wicked problem. Solutions to such problems generate waves of consequences, and it's impossible to know how these waves will eventually play out.
- Every implemented solution to a wicked problem has consequences. Once the Web site is published or the new customer service package goes live, you can't take back what was online or revert to the former customer database.
- Wicked problems don't have a well-described set of potential solutions. Various stakeholders have differing views of acceptable solutions. It's a matter of judgment as to when enough potential solutions have emerged and which should be pursued.
- Each wicked problem is essentially unique. There are no "classes" of solutions that can be applied to a specific case. As Rittel and Webber wrote in "Dilemmas in a General Theory of Planning," "Part of the art of dealing with wicked problems is the art of not knowing too early what type of solution to apply."
- Each wicked problem can be considered a symptom of another problem. A wicked problem is a set of interlocking issues and constraints that change over time, embedded in a dynamic social context.
- The causes of a wicked problem can be explained in numerous ways. There are many stakeholders who will have various and changing ideas about what might be a problem, what might be causing it and how to resolve it.
- The planner (designer) has no right to be wrong. Scientists are expected to formulate hypotheses, which may or may not be supportable by evidence. Designers don't have such a luxury—they're expected to get things right.
Recognizing the Beast
What kinds of problems are wicked? Locating a new freeway or homeless shelter;
optimizing all the features on a new model car; choosing the best way to reengineer
a business process—all these examples are challenges that make you deal
with something new, with change to an existing design or with different ideas
about the way that change should take place.
How do you identify a wicked problem? First, look for divergence. If requirements are volatile, constraints keep changing, stakeholders can't agree and the target is constantly moving, you're probably dealing with a wicked problem. If considerable time and effort has been spent, but there isn't much to show for it, wickedness is probably lurking nearby. And if business process reengineering is involved, you stand a good chance of encountering a wicked problem.
Lambs vs. Lions
According to Rittel and Webber, the opposite of a wicked problem is a "tame"
one. Tame problems may be quite complex, but they lend themselves to analysis
and solution by known techniques. A traditional linear process (for example,
a common project management process) can produce a workable solution to a tame
problem in an acceptable period of time, and it's clear when that solution has
been reached.
The most fundamental rule for handling wicked problems is that they must not be treated like tame problems. Though at first you may think you've calmed the beast with standard techniques, its volatile nature will eventually surface in the form of changed constraints, varying requirements or stakeholder resistance. Beware: If you treat that lion like a lamb, the beast will soon become your own personal nightmare. If you've ever had customers sign off on a requirements document, and then implemented it exactly as agreed, only to have the users reject the system as not meeting their needs, you've been there.
In "Dilemmas in a General Theory of Planning," Rittel and Webber explain that "The classical systems approach … is based on the assumption that a … project can be organized into distinct phases: 'understand the problem,' 'gather information,' 'synthesize information,' 'work out solutions' and the like. For wicked problems, however, this type of scheme does not work. One cannot understand the problem without knowing about its context; one cannot meaningfully search for information without the orientation of a solution concept; one cannot first understand, then solve."
Wicked problems can best be elucidated through discussion. Consensus develops through the process of exploring alternative approaches to the problem, competing interests, priorities and constraints. The application of more formal analysis tools is impossible before the problem can be articulated in a concise, agreed-upon, well-bounded manner. In other words, the problem must be tamed before it's tackled.
Death Marches and Other Diversions
Wicked projects arise when a project is organized to tackle a wicked problem
as if it were tame—thus creating a monster. One potential form of this
Frankenstein is the "death march project"—as defined by OO guru
and Cutter Consortium chairman Edward Yourdon in his book, Death March Projects:
The Complete Software Developer's Guide to Surviving "Mission Impossible"
Projects (Prentice Hall, 1997), a mission-critical project with less than
half the time and resources necessary for completion. Think about it—death
march projects often arise when volatile requirements, changing constraints
and stakeholder disagreements meet up against immovable deadlines. They're frequently
a symptom of management's unwillingness to tackle wicked problems.
Failure to recognize and deal with wicked projects has given software development a bad name. A 1994 Standish Group Report found, for example, that about a third of software development projects get canceled, and half do not meet their original cost projections. Some have taken this to indicate that the state of software development is in disarray. However, it can also point to the large number of development project teams out there, trying to address wicked problems with the wrong approach.
When a project fails, typically, management concludes that the organization is immature, and thus aims for more maturity—more "by-the-book" project management, for instance: send project managers to certification classes; form a project office to develop a standard process. This usually means imposing more requirements documentation, more analysis, more planning and tracking against the plan. Managers feel that further implementation of the classic project-management processes will avert future disasters—and, if the failed project addressed a tame problem, this might be the case.
However, classic project-management practices simply do not work with wicked projects. In fact, according to Rittel and Webber, attempting to baseline requirements and then use an analytical approach to deal with wicked projects is a recipe for disaster.
Whipping Wickedness
Step 1: Identification. The first step in combating most systemic
problems is to recognize them for what they are. If the project must satisfy
stakeholders who don't agree on its fundamental issues, you've ventured into
wicked territory. Certainly, in a perfect world, there should be an executive
sponsor, and the customers should speak with one voice—but what if they
don't? Despite a semblance of consensus, chaos may loom beneath the surface,
and you'd better be prepared for it. Let's face it: This is a wicked project,
and everyone will be better off when it's recognized as such.
Stakeholders who don't know what they want are rampant in wicked projects. This is the IKIWISI (I'll know it when I see it) syndrome, and it's pretty common. Suppose you're operating in a contractual environment that specifies that a requirements document must be signed off before proceeding with development. What if your users, deep down, suffer from the IKIWISI syndrome? You have a wicked project, no doubt about it.
Contracts impose an extra dimension on projects, which often creates conflict between the stakeholders' real needs and the contractual requirements. In a time of rapidly changing technology, changing stakeholders, changing competition and a changing economic environment, contractual requirements tend to become obsolete. The tension created when contractual requirements don't meet current needs slows down the project at best, and at worst makes it a truly wicked project.
Step 2: Taming. In his recent article, "Inside Microsoft: Balancing Creativity and Discipline" (Harvard Business Review, Jan. 2002), Robert Herbold gave an excellent example of taming a wicked project. As Microsoft's new COO in 1994, he realized that he had to impose centralized financial, purchasing and human resource systems on staunchly independent Microsoft business units. Here's how he did it:
Executive Support. Bill Gates made it clear to all divisions that anyone not reporting results through the new system would be deemed to have no results at all. Steve Ballmer told balking general managers: "We define the measures. You put all your creative juices into growing them."
Clear Problem Definition. All central systems were implemented with off-the-shelf software that dumped data into a data warehouse, so the problem was constrained by the capabilities of commercially available software. There were few system decisions to be made beyond which package to use and how to configure it. This was accomplished by a small group of technical and business experts because, according to Herbold, "You can get the job done in the time it takes a cross-functional committee to decide on its goals."
Separation of Concerns/Reduction of Stakeholders. Business units had Web-based access to the data warehouse for each system, and they could get at the data however they wanted to, but they didn't have access to the packaged software. Thus, business unit managers had no stake in the packaged software, and where they did have an interest, which was data access, they had a high degree of control and flexibility. This reduced the number of central-system stakeholders to a minimum.
Appropriate executive support is the most crucial aspect of taming wicked projects; they can't easily be tamed at a low level. This means that the sponsor must recognize wicked problems and act accordingly. If the sponsor can't resolve conflicts and the customers don't speak with one voice, the project can't be tamed.
Step 3: Adapting. Most wicked projects can't be tamed; after all, the easy problems have been solved. If you have a wicked project and it becomes clear that no amount of executive support is going to make things any easier, it's time to use an adaptive process.
An adaptive process is one that begins with a general target, and uses feedback to refine both the target and the process as learning takes place. For instance, when bringing up a new adhesive tape manufacturing line, a series of designed experiments, along with adaptive process control algorithms, will be used to establish and continually adjust pressures, temperatures and line speeds.
There's nothing difficult about adaptive processes—they've been used successfully in world-class product-development organizations for decades. Most public and business policy is developed in this manner, as is most marketing—and all sophisticated process-control systems use adaptive control techniques.
In software development, wicked problems must be resolved through discussion, consensus, iterations and acceptance of change as a normal part of the process. Just as in process control, you start out with a general description of the target and use a series of iterations, customer feedback and adjustment to refine the target. In order to use adaptive control for software development, however, you have to accept that you can't freeze cost, schedule and features at the same time. One of these has to be adaptable. If the schedule is firm, you have to accept that either the resources or (better) the feature list are a variable.
Because adaptive processes cater to situations in which it's not possible to fix all three variables of cost, schedule and features at the beginning of a project, there is often intense pressure not to use these processes. Don't bow to it. In fact, when used for wicked problems, adaptive control techniques are far more successful at producing desirable outcomes than are traditional project management techniques. Microsoft, for example, develops all of its software using adaptive techniques.
In "How Microsoft Makes Large Teams Work Like Small Teams" (Sloan Management Review, Fall 1977), Michael Cusumano describes Microsoft's software development strategy: "Focus creativity by evolving features and 'fixing' resources." Microsoft controls the scope of a project with a brief vision statement containing a prioritized list of features. It limits resources by enforcing release deadlines. It manages integration with daily builds. Developers are paired with testers expected to use coding standards. The schedule has a 20-50 percent buffer, and delivered features are expected to vary from plan by more than 30 percent.
Scrum
Scrum is an excellent adaptive project-management approach for wicked software
development projects. This rugby term was first used by business management
analysts H. Takeuchi and I. Nonaka in their article, "The New New Product
Development Game" (Harvard Business Review, Jan.–Feb. 1986). Considering
that a new product–development project is by its nature a wicked problem,
the best new product–development processes present a good model for solving
other similar problems.
Ken Schwaber and Mike Beedle describe the use of Scrum for software development projects in their book, Agile Software Development With Scrum (Prentice Hall, 2001). The Scrum process is based on incremental action that creates a basis for stakeholder dialog and project feedback. A Scrum project collects stakeholder input in a feature list called "the backlog." Each month, the development team starts at the top of the backlog and selects as many of the top-priority features as they think can be developed in the next 30 days. The team is then left alone for the month, at the end of which time, the result is demonstrated to the stakeholders. This provides a basis for rethinking the backlog features and priorities. The stakeholders are allowed to modify and reprioritize the backlog, after which the team selects its next month's work from the top of the list.
The Scrum process allows the development team to make regular progress, even if their problem is not well understood. At the same time, it offers a method for stakeholders to discuss the problem and reach consensus, creating a way for solutions to emerge, as is necessary for wicked problem resolution.
At the same time, the Scrum process provides a high degree of predictability. Each line on the backlog is estimated, and the estimates are added to create an overall project completion estimate. After three months, a graph of the backlog estimate against time is a highly reliable indicator of the project's progress and estimated completion date. Features may be added or subtracted from the backlog monthly to adjust for constraints, as well as changing stakeholder interests. The trend in the backlog estimate is a reliable indication of whether the project is progressing according to plan.
Large Projects
Another approach to adaptive project management, particularly on large projects
that replace legacy systems, is described by ThoughtWorks' Matt Simons in "Big
and Agile?" (Cutter IT Journal, Jan. 2002). Simons suggests starting with
a single thread through a system. For example, in an insurance application,
implement a new policy, renewal and cancellation thread. The thread must be
limited in some way—for instance, the initial system will handle only a
specific policy type. It should be limited enough to make it possible to go
live with the thread in about three or four months. Implementing a thread, rather
than the entire system, keeps the scope of the first learning experience within
acceptable risk boundaries.
By selecting a thread, several small teams can work on various system modules at the same time—initial policy qualification, renewal notices or financial system interfaces, for example. The secret of large agile projects, as both Microsoft and ThoughtWorks demonstrate, is parallel development coupled with constant integration. Daily builds with automatic testing keep the small teams synchronized, while allowing them to proceed more or less independently.
Once the first thread is successfully in production, a large measure of the learning has taken place and much of the uncertainty has been removed from the project. The remaining implementation is a variation on the theme. Simons advocates implementing by thread, rather than by module, if at all possible, because this is the fastest way to tame the project.
Beyond Tradition
If software development projects have not responded well to traditional project
management practices, the fault may lie in the use of inappropriate methodologies.
Wicked projects, rife with conflict in stakeholder requirements and changes
in management constraints, are best served by an adaptive process instead of
traditional methodologies. At minimum, adaptive project management practices
will quickly determine if a proposed solution will be successful, and thus provide
valid data for project portfolio management.
