Measuring Maturity

Software Development

In the mid ’90s, a company called Zeos assembled PCs in my home state of Minnesota. I was impressed when Zeos was named as a finalist for the Malcolm Baldrige National Quality Award, because competing for this honor is the equivalent of a software company trying to reach CMM Level 5. At the same time that Zeos was focusing on the Malcolm Baldrige Award, a similar company in Austin, Texas, named Dell Computer, was devoting all of its energy to two rather different objectives: Keeping inventory as low as possible, because it’s the biggest risk in the PC assembly business, and delivering computers as quickly as possible after customers decide what they want. Dell couldn’t possibly meet these goals unless it had the key Malcolm Baldrige capabilities (planning, execution, analysis and process management) in place, but the award wasn’t its priority. On the other hand, to be a finalist, Zeos had to spend large amounts of executive and management attention on Malcolm Baldrige certification. Which of these two companies would you call the most mature?

A company doesn’t rise to the top of its industry by perfecting its prescriptive procedures. While General Motors was busy refining its four-phase process for product development, Toyota and Honda were developing cars in nearly half the time, at half the cost. The resulting cars captured a much larger market share. Yet, to Detroit’s eyes, the Japanese development approach was decidedly immature—why, they even let die makers start cutting dies long before the final drawings were released!

All too often, maturity is viewed as documenting and following a detailed, step-by-step process. This may work for simple things, but when a network of complex interactions is involved, just a bit of variation wreaks havoc with a prescribed process. The ensuing unpredictable results are usually met with heroic attempts to reduce the variation and further standardize the process. Dell and Toyota “thought different”: Accepting variation as inherent in their process, they designed processes intelligent enough to capitalize on that fact, while their competition was muddling with the prevailing notion of how a “mature” company should manage things.

Most managers have a mental model of what it means for an organization to be “mature” or “world class.” Unfortunately, these mental models can block vital paradigm shifts destined to reshape their industry. This has happened in manufacturing and product development. Can it be happening in software development today?

Miller’s Law

Maturity models such as CMM attempt to certify that a large number of “good practices” are in place, faithfully followed everywhere in the organization. However, it’s easy to lose sight of the forest for the trees. Because we humans can hold only a few chunks of information in short-term memory at once (Miller’s law claims that this number is 7 plus or minus 2), we tend to decompose difficult concepts into parts, so we can deal with one part at a time. There’s nothing wrong with this natural tendency, except that we often forget to reconfigure the parts into a whole and take their interactions into account.

Unfortunately, decomposition works only if the whole is indeed equal to the sum of its parts, and interactions between the parts can be ignored. In practice, this is rarely the case; in fact, optimization of the parts tends to suboptimize the whole. For example, a manager might think that the best way to run a testing department is to ensure that every single person is working all of the time. To guarantee each individual’s maximum productivity, he makes sure there’s always a pile of testing waiting to be done. Down the hall, the operations manager knows better. She understands that if she runs the servers at high utilization, the entire system bogs down, just like rush-hour traffic. If only the testing manager would realize that his policy of fully utilizing testing resources creates the same kind of traffic jam in software development.

Decomposition of a problem into pieces is a standard problem-solving heuristic, but it needs to be accompanied by regular aggregation of the parts into a whole. It’s here that iterative development shines, because it forces us to develop, test, integrate and release complete threads of the system early and often. Thus, we reset our view of the big picture every iteration. However, iterative development requires that we decompose the problem space along dimensions that are orthogonal to those traditionally recommended. Instead of decomposing the problem into requirements, analysis, design, programming, testing, integration and deployment, we decompose the problem into features along the natural fault lines of the domain.

Decomposition vs. Abstraction

There is an alternative: Rather than decomposing a problem to a lower level, you can abstract it to a higher level. In both cases, you’ve reduced the number of things you need to think about to 7 plus or minus 2. But when you move up the problem to a higher level of abstraction, interaction of the parts is maintained, so abstractions would seem to be the better approach to solving complex problems.

There’s a catch: People can’t create abstractions without understanding the domain, because correct abstractions require that the abstractor knows what’s important, what fits together naturally, and where the domain’s natural joints or fault lines are. For this reason, domain experts tend to abstract to a higher level, while novices tend to decompose the problem—and that decomposition probably won’t coincide with the domain’s natural fault lines.

When we decompose a problem in a manner that doesn’t fit the domain and then drill down to the details too fast, important things are overlooked. It’s tempting to think that a lot of early detail will help us find all the hidden gremlins that might bite us later. This is true only if those gremlins are actually inside the areas we investigate, but the tough problems usually lurk between the cracks in our thinking. Unfortunately, we discover those problems only when we integrate the pieces at the end, and by that time, we’ve invested so much in the details that change is very difficult.

Instead, it’s safer and more efficient to work from a higher level of abstraction, but only when we don’t drill down to detail right away. Taking a breadth-first approach, we keep options open and gradually fill in the details. In this way, we’re more likely to find those gremlins while we can still do something about them.

Assessment In Lieu of Certification

CMM is often implemented as a certification process, decomposing “maturity” into separately verifiable capabilities. The danger in this definition is that we may lose sight of the forest for the trees. For example, focusing on requirements management can be detrimental to giving users what they really want. Focusing on quality assurance separate from development can destroy the integrating effect of developers and testers working side-by-side every day. Focusing on planning as a predictive process can prevent us from using it as an organizing process.

For a better approach to determining an organization or individual’s underlying competence, use assessment rather than certification. Assessments present challenging situations that can be successfully navigated only with a host of capabilities; if the challenge is met, the presence of these capabilities can be inferred. For example, a pilot’s ability to fly a plane can be assessed by observing how the pilot lands the plane in a stiff crosswind. One DBA I knew demonstrated her capability when she correctly diagnosed a database lockup problem that had shut down production.

Consider your hiring process. When a candidate lists Microsoft or PMI certifications, take that into account, but seek a successful track record that demonstrates that the certifications were put to good use. One software company I know gives job applicants a one-hour logic test. Though it doesn’t have a line of code, the test does an admirable job of assessing the candidate’s capability to think like a good developer.

Measure Up

Measurements are funny things. Over time, you’ll get what you measure, so you must be careful to measure everything that’s important. The trouble is, it’s difficult to measure everything, and when we notice that something’s missing, we tend to add another measurement to plug the hole. A better approach to dealing with gaps in a measurement system is to reduce the number of measurements and raise the span of each measurement.

In CMM, each key process area (KPA) addresses something that caused a failure of some software project somewhere. For example, when code isn’t kept under version control, all sorts of problems result. So one KPA requires using a good source-code repository with version control. As people discovered new ways for software projects to fail, KPAs were added to address the new failure modes. Nevertheless, all of those KPAs still don’t cover everything that could go wrong.

Once we admit that we can’t measure everything, we can move from certification-style measurements to assessment-style measurements. For example, in project management, we decompose the measurement system into cost, schedule, scope and defects, and we try hard to make these measurements work because we think they’re the measurements we should use. But no matter how hard we try, we’re often unsuccessful in measuring true business value in this way, because of decomposition’s lack of holistic view.

What if we just measured business value instead of cost, schedule, scope and defects? When I was developing new products at 3M, we didn’t pay much attention to cost, schedule, scope or defects. Instead, we developed a profit and loss (P&L) analysis that was used to check the impact of a late introduction date or a lower unit cost. The development team tried to optimize the overall P&L, not any one dimension. It may seem strange that a project team would concern itself with the business’s financial model, but I assure you that it’s a far better decision-support tool than cost, schedule, scope and defects.

The Measure of Maturity

Our industry could use a simpler measurement of software development maturity, one that isn’t subject to the dangers of decomposition and doesn’t attempt to defy Miller’s Law with a long list of items to worry about. I propose that we use a measurement that has been successfully employed in countless organizations as a real-world indicator of the kind of capabilities measured by CMM. To introduce this measurement, let’s go back to Dell, which measured two critical elements: the level of inventory throughout the entire system, and the speed with which the organization can repeatedly and reliably respond to a customer request.

“OK,” you may say, “that’s fine for manufacturing, but software development is different.”

Yes, but the fact is, these measurements still work.

The level of inventory in your system is the amount of stuff you have under development. The more inventory of unfinished development work you have sitting around, the greater you’re at risk of it becoming obsolete, getting lost and hiding defects. If you capitalized it, you also bear the risk of having to write it off if it doesn’t work. The less of that kind of stuff you have on hand, the better off you’ll be.

The speed with which you can respond to a customer is directly proportional to the amount of unfinished development work you have clogging up your system. In truth, the two measurements above are inextricably linked: You can deliver faster if your system isn’t clogged with unfinished work.

Shortly after Zeos won the Malcolm Baldrige award, the company was acquired by Micron. Once neck and neck with Gateway and Dell, Micron—and its acquisition—have long since exited the business. Zeos’s sad fate demonstrates that great processes in themselves don’t offer much of a competitive advantage. Taking a different tack, Dell focused on eliminating the need to forecast. And, instead of improving its ability to suppress changes, Toyota learned how to develop a product without freezing design until just before it went into production.

The assessment approach to maturity entails rapid, reliable, repeatable delivery on customer needs. You can’t reliably and repeatedly deliver quickly if you don’t have version control, built-in quality, and ways to discover requirements quickly and routinely translate them correctly into code. But reliable, speedy satisfaction of customer needs requires far more than individual capabilities; it demands a flexible process that welcomes, rather than battles, the inevitable, irresistible force of change.