Examining the Software Development Process
How do you measure the effectiveness of a development process?
James O. Coplien
Jim is a member of the technical staff at AT&T Bell Labs and the author of Advanced C++ Programming Styles and Idioms (Addison-Wesley, 1992). Jim can be contacted at [email protected].
From both an applications and a systems perspective, software is getting larger and larger, involving more and more programmers during the development cycle. For reasons ranging from resource planning to cost control, it is becoming increasingly important that you be able to evaluate and control the overall software-development process.
At Bell Laboratories Software Production Research Department, we've borrowed from object-oriented analysis a tool called "CRC cards" to evaluate development processes. CRC is an acronym for "classes, responsibilities, and collaborators"--three of the most important dimensions of abstraction in object-oriented analysis. (For more information on CRC cards, see Kent Beck's "Think Like An Object," UNIX Review, September 1991.) Our intent was to extend the use of CRC cards--originally designed to analyze system-software architecture development--to study software-development organizations within AT&T. Later, we applied the technique to other development projects (including Borland's Quattro Pro for Windows) to test the validity of the methodology and to learn from other organizations.
When it comes to capturing the essential properties of organizational roles, we've found that CRC cards have advantages over traditional role-modeling tools. CRC cards are informal, easily learned and understood by developers, and delineate the social interactions that are important for sound, empirical models of organizations such as software-development groups. CRC cards let us gather process and organizational information from our development colleagues using their vocabulary, at their level, giving us a faithful model of a development culture.
To understand, compare, and model the software-development process, we enter CRC-card data into a process-evaluation program called "Pasteur" which contains transcription of the CRC cards wrapped in a hypertext database called "Eggs." (See Hypertext: Concepts, Systems and Applications, Cambridge University Press, 1990.) The Pasteur environment lets the programmer create an on-screen card abstraction, then type information into appropriate fields. Cards can be resized and moved about the screen. On a typical workstation display, we can fit about 40 full-size, nonoverlapping cards. Once data is entered, we can browse, cluster, and animate the cards.
CRC cards provide an object-oriented analysis of the structure of an organization by dividing it into objects ("roles") that are cohesive locales of related responsibilities. Each role's responsibilities to the organization are written on the left side of a 3x5 index card; see Figure 1. The right side of the card lists the helpers (or collaborators) used by a role to carry out its responsibilities. Responsibilities and collaborators are discovered in a real-life, role-playing exercise where development scenarios are simulated. The interests of a role are represented by someone who commonly fills the role, by a domain expert in the appropriate area, or by someone who is otherwise familiar with the work.
Collaborations between roles form a network, or graph. The edges of the graph are subjectively weighted by the participants (high, medium, or low) according to the strength of the coupling between two roles. The graph can be visualized in many different ways: a force-based network, a topologically sorted hierarchy or directed graph, an interaction grid, and so on. We use the Pasteur process-analysis environment to create and interact with such visualizations.
These visualizations offer insights into organizational dynamics. For example, cliques can be identified from the natural-force-based networks. Interaction grids offer insight into the cohesiveness of an organization. Highly specialized patterns have been noted in visualizations using each of these techniques, including a tendency for roles to cluster according to their degree of extrovertedness or introvertedness with respect to the process community.
Once data has been gathered, analysis can begin. One obvious analysis is to illuminate all the roles having a strong coupling to a chosen role to determine the centrality of that role in the process. A role was considered central to the process to the degree it had strong coupling with remaining roles in the process. One curiosity we discovered was that the "developer" role had strong coupling with most of the internals of the software-integration process. This was surprising, since the developer is supposedly isolated from the details of software integration.
We have found many other recurring patterns in these visualizations, many of which bear out common management folklore, others which are counterintuitive, and still others which deserve further study. One goal of our research is to correlate these patterns to high productivity, quality, and responsiveness in the organizations that generate them. By understanding these patterns, we hope to develop principles from which new, highly productive organizations can be built. This work follows the lead of the generative-pattern movement which, like CRC cards, started as a software-design phenomenon but has also found a home in organizational analysis (see "Patterns and Software Development" by Kent Beck, DDJ, February 1994).
Applying the CRC-Card Process in the Real World
The development of Borland's Quattro Pro for Windows spreadsheet (since sold to Novell) is one of the more remarkable processes we've encountered in the Pasteur process-research project. The project assimilated requirements, completed design, implemented one million lines of code, and completed testing in 31 months. Coding was done by no more than eight people at a time, which means that individual coding productivity was higher than 1000 lines of code per staff-week. (Granted, lines of code is an imperfect measure of productivity at best. However, a disparity of one or two orders of magnitude between the Borland experience and more typical numbers from the rest of the industry cannot be explained with the usual attacks on source-line measurements.)
The project capitalized on its small size by centering development activities around daily meetings where architecture, design, and interface issues were discussed. Quality-assurance and project-management roles were central to the development sociology, in contrast to the developer-centric software production most often observed in our studies of AT&T telecommunications software. Analyses of the development process are "off the charts" relative to most other processes we have studied.
As with all Borland software, Quattro Pro for Windows (QPW) was designed to be an independent, self-contained deliverable, sharing a common infrastructure and look-and-feel (and, conjecturally, code providing this functionality). The total code volume of all Borland software, expressed as original source lines, is huge: tens, if not hundreds, of millions of lines of code (my estimate).
QPW had a core team of four people who interacted intensely over two years to produce the bulk of the product. Prototyping was heavily used: Two major prototypes were built and discarded (the first in C; the second, called "pre-Crystal," in C++). The team defined the architecture, built early prototypes and foundation code, and participated in implementation through its delivery. Additional programmers were added after about six months. The prototypes drove architectural decisions that were discussed in frequent (almost daily) meetings.
The methodology was iterative. Except for architectural dialogue, the developers worked independently. Early code can be viewed as a series of prototypes that led to architectural decisions, and drove the overall structure of the final system.
The QPW final implementation stages stressed Borland's C++ compiler--being developed in parallel with QPW--to the max. There was uncharacteristically tight coupling between the QPW group and the language group. QPW was one of the largest and earliest projects for the C++ compiler.
As soon as the software took shape (after about a year), additional roles were engaged in development activities. Quality assurance (QA), testers, and others were at last allowed to see and exercise copies of the code that had been kept under wraps during early development. These roles had been staffed earlier, but were engaged only when the developers felt they had something worth testing.
Analysis of the Pasteur Data for QPW
We most frequently use a natural-force-based network analysis to analyze organization data collected in the Pasteur database. This analysis produces an adjacency diagram in which a default repelling force is established between each pair of roles. There is also an attracting force between pairs of roles that are coupled to each other by collaboration or mutual interest; a stable placement occurs when these forces balance. Figure 2 shows the diagram that results when applying this analysis to QPW. Several items set this project apart from other organizational process models we've made:
- The QPW process has a higher communication saturation than 89 percent of the processes we've looked at.
- The adjacency diagram shows that all roles have at least two strong connections to the organization as a whole. The project's interaction grid is dense. The coupling per role is in the highest 7 percent of all processes we've examined. This is a small, intensely interactive organization.
- There is a more even distribution of effort across roles than in most other processes we've examined. The roles in the adjacency diagram are shaded according to their intensity of interaction with the rest of the organization. In the QPW process, project manager and QA glow brightly; coders, a little less so; architect, product manager, and beta sites are "third-magnitude stars;" and tech support, documentation, and VP still show some illumination. Most "traditional" processes we've studied show a much higher concentration of interaction near the center of the process. That is, most other processes comprise more roles that are loosely coupled to the process than we find in QPW. That may be because QPW is self-contained, or because it is small. It may also be because the process was intense: a high-energy development project racing to hit an acceptable point on the market-share curve.
- Project manager and product manager are tightly coupled, central roles in the process. These managerial roles were filled by individuals who were also key technical contributors to the project (they wrote real code), which contributed to their acceptance and success as process hubs. Product manager was a role that was employed only after a year of development.
- QA is a tightly coupled, central role. Many organizations consider QA to be an external function, outside their organization and process. At Borland, QA becomes a way of life once development has converged on a good design and a stable user interface. For QPW, this was about 12 months into development.
- The CEO (Philippe Kahn) figures strongly in the organization.
Each project member had to personally sign off on a set of project floppy disks before they were released to the next stage (beta test or the "street"). Accountability, ownership, and pride in one's work were central to the process.
QPW is organized along lines of domain specialization. Domains important to QPW are dependency-registration software, human interfaces, databases, and a few others, and an individual was identified for each of those domains. Within a domain, each individual did what he or she was good at. Equally important is what these individuals were not good at; they were not expected to take responsibility for domains not related to their specialty. For example, when it came to documentation, developers were supported by a documentation team that developed internal and external documentation. The time spent by developers in conveying information to the documentation organization is far less than it would take for them to commit it to writing, put it into an acceptable format, and have it edited for linguistic elegance. (By contrast, most AT&T developers write their own memos. It's not clear whether this stems from our history, our organizational boundaries, the nature of our business, or reward mechanisms. In any case, developers spend roughly 13 percent of total development time creating and refining memos.)
QPW development was highly iterative. To understand the nature of the iteration, you must understand its ramifications for architecture and implementation. You must also understand the culture in which changes were approved and decisions made. This takes us into the realm of project meetings, always a topic of interest in a large development organization.
The core architecture team met daily to hammer out C++ class interfaces, discuss overall algorithms and approaches, and develop the underlying mechanisms on which the system would be built. These daily meetings lasted several hours; from what I heard, the project was made more of meetings than anything else. Everyone's external interfaces were globally visible and globally discussed. The software structure was built on the domain expertise brought to the table by domain experts, but it was socialized and tempered by the needs of the product as a whole.
In spite of the intense, meeting-oriented development culture, class implementations were fleshed out in private. Individuals were trusted with doing a good job of implementation: After all, project members were acknowledged experts in their domains.
There are three observations worth noting about the QPW organization's communication architecture:
- Meetings are not a bad thing.
- Development took place on two levels: architecture and implementation. Both were ongoing and interacted with each other strongly. New implementations suggested architectural changes, and these were discussed at the daily meetings. Architectural changes usually required radical changes to the implementation. The implementors' ability to quickly reflect those changes in their implementation was essential.
- The development interaction style was a good match for the implementation technology the group had selected. Object-oriented development leads to abstractions whose identity and structure are largely consistent across analysis, design, and implementation. Classes hide implementations and localize design decisions, though their external interfaces are globally visible. Mapping C++ classes and people close together made it possible for developers to reason about the implementation off-line, away from the meetings that dealt with interface issues.
A question frequently posed to organizations using iterative techniques is: "How do you mark progress or do scheduling?" For QPW, the answer had two parts: First, the team relied on its experience in sizing similar jobs, and found the overall estimates to be satisfactory. Second, they kept multiple sets of books internal to Borland to achieve different goals. The hardest of the dates was owned (and not divulged) by the financial group. A "real" street date was needed so the company could provide planning and resource support to the development. But internal scheduling provided incentive, focus, and pressure for development to move ahead.
Looking Inward
QPW used iteration throughout the development cycle, increasing the stability of the software and decreasing iteration over time. This iteration took place in what might be described as a "traditional corporate context." From its outset, QPW was a strategic, visible product in the company. That meant that all development areas were primed for its deployment, including QA, documentation, and product management.
Though these areas were staffed from the outset, they were denied access to the details of the product until about a year into development. That gave the architect/developers room to change the functionality, interface, and methodology of the project before interfacing it with the corporate culture and ultimately with users.
Can an organization without an explicit, conscious process effort enjoy the same process benefits as an organization with full ISO 9000 process certification? Certified organizations may reap stronger process benefits than those lacking any formal concern for process; nevertheless, this Borland project had many of the hallmarks of a mature development organization.
Borland is not subject to the ISO 9000 series process standards, has no concept of its SEI CMM rating, and is not conversant with the software-development-process lingo being used increasingly in large software organizations. For someone interested in "process" to visit was a rare event. Before going through the CRC-card exercise, my presence as a process engineer was viewed with interest, curiosity, and even suspicious doubt. By the time I left, those involved were able to identify some parts of their value system and culture with what we call "process."
Even though the organization had no codified system of process, it was keenly aware of what it did, how it did it, and what worked. It viewed software development as something fundamentally driven by special cases (at least for initial, generic development); repeatability was not an important part of their value system. Members of the organization were nonetheless able to articulate, in great detail, aspects of their process that demonstrated that they shared a single model, perhaps based on development rules, of how development should be done.
Many organizations we've interviewed have a weak or confused notion of the responsibilities and interaction of roles within the organization. Most AT&T organizations with a weak notion of process are those who have not gone through an ISO audit, yet developers' notions of their roles even in some ISO-certified organizations are fuzzy at best. Other organizations that do not have any conscious process culture are still able to articulate their process in explicit terms, at a level of abstraction that transcends technology, tools, or methodology.
In his book, Quality Software Management, Vol. 1 (Dorset House, 1991), Gerry Weinberg describes several levels of organization. Organizations at levels 1 and 2 need strong managerial direction. There is a paradigm shift between levels 2 and 3 of the SEI Capability Maturity Model (CMM), so organizations at level 3 and above are self directing. Borland appears to be in this latter category--though it may not register a level-3 rating according to commonly accepted criteria.
Charlie Anderson, one of the QPW architects, told us how the project team felt about itself and its accomplishments. "We are satisfied by doing real work," he noted as he thought about how the project dovetailed daily architectural meetings with implementation. "Software is like a plant that grows," he mused. "You can't predict its exact shape, or how big it will grow; you can control its growth only to a limited degree."
Process and Quality
One widely held stereotype of companies that build PC products (or of California-based companies) is that they hire "hackers" and that their software is magic, unreadable spaghetti. Meeting with the QPW group dissolved that stereotype for me. Their constant attention to architectural issues, efforts to build an evolving structure, and care to document the system well (both externally and internally), are all hallmarks of professionalism.
If there was any disappointment on the project, it was in the inability to bring down the bug curve as fast as they wanted. They noted that the shapes of software-development bug curves are well known, so there is hope of predicting how long it will take to ferret out an acceptable fraction of the remaining errors. However, the boundary conditions for the curve aren't known at the outset, so it is difficult to predict the exact shape of the curve until some bugs have been discovered and resolved. Inability to predict the exact shape of this curve resulted in a modest schedule slip.
Conclusions
Can other organizations capture the architecture of the Borland development process? To the extent that large jobs can be partitioned into small ones, the Borland approach may be suitable for individual parts of large developments.
Borland was able to coax a lot of production code from a few people in a short time. Perhaps a PC-based development environment and PC-based deployment platform make developers more effective, and perhaps QPW doesn't have the same fault-tolerance requirements one finds in large telecommunications systems. However, those considerations alone don't seem to account for figures that are orders of magnitude above industry norms.
Figure 1 Typical CRC card.
Figure 2 Natural-force-based analysis of the QPW project roles.
Figure 3 Interaction grid for the QPW project.
Copyright © 1994, Dr. Dobb's Journal
