Using the Cognitive Dimensions Framework
While we tend to use the cognitive dimensions framework to analyze the results of a usability study, you don't necessarily need to go to such lengths to get usability feedback on your own API. You can use the framework to analyze a set of code samples before the API has even been implemented.
For example, consider a fictional API for building e-mail clients. Example 2 demonstrates how to use this API to create a new e-mail message and send it.
IPAddress hostIPAddress = IPAddress.Parse("130.23.43.234");
IPAddress theAddress = new IPAddress(hostIPAddress);
CEmailServer theServer = new CEmailServer (theAddress);
if (theServer.Connect ())
{
CMailbox theMailbox = theServer.GetMailbox ("stevencl");
theMailbox.Open("stevencl", true);
CEmailMessage theMessage = theMailbox.NewMessage();
theMessage.Build("message", " subject", "[email protected]");
theMailbox.Send (theMsg, true);
theMailbox.Close();
}
else
Console.WriteLine ("Could not connect to the mail server");
Example 2: Using an API to create/send a new e-mail message.
Developers using this API need to create instances of IPAddress, CEmailServer, CMailbox, and CEmailMessage. Furthermore, the CMailbox and CEmailMessage are created by calling factory methods exposed by the CEmailServer and CMailbox classes, respectively. Developers using this API need to create and configure all of these objects in conjunction with one another.
To review this API in terms of the cognitive dimensions framework for this given scenario, we ask a set of questions for each dimension. For example, looking at the work-step unit, we might ask:
- Does the amount of code required for this scenario seem just about right, too much, or too little? Why?
- Does the amount of code required for each subtask in this scenario seem just about right, too much, or too little? Why?
For progressive evaluation, we might ask:
- How easy is it to stop in the middle of the scenario and check the progress of work so far?
- Is it possible to find out how much progress has been made? If not, why not?
Or, for role expressiveness dimensions:
- When reading code that uses the API, is it easy to tell what each section of code does? Why?
- Are there some parts that are particularly difficult to interpret? Which ones?
- When using the API, is it easy to know what classes and methods of the API to use when writing code?
By answering these questions, it is clear that there are a number of issues that arise related to the work-step unit, progressive evaluation, and role expressiveness.
Considering the work-step unit dimension, it's clear that developers must complete significant chunks of work with this API when they want to implement simple steps such as create an e-mail message or open a mailbox. For example, in the just mentioned API, you can't write code to create an instance of a CEmailMessage without writing all of the surrounding code to connect to a server, retrieve a mailbox, and open the mailbox. When the chunks of work are large enough, you have to use mental resources to keep track of the work that has been done and the work that remains to be done.
It is difficult for developers using this API to stop at some point before completing the scenario to check their progress. In other words, the API does not support progressive evaluation. Because developers cannot create an instance of a CEmailMessage without connecting to a server and opening a mailbox, there is no way for them to isolate the code they need to write to configure an e-mail message. The lack of progressive evaluation exposed by this API means that runtime errors can't easily be tied to specific lines of code or objects and makes debugging code written against the API more difficult. It also makes learning how to use the API difficult for some users because it is not really possible to explore and learn about individual types on their own.
Lastly, the role expressiveness is poor. There are at least three lines of code that are difficult to understand. Let's look at the first one:
CEmailServer theServer =
new CEmailServer (theAddress);
Does this create a new mail server or an instance of the class representing an existing mail server? It's actually the latter, but the fact that we are creating a new instance of a class to represent an object that already exists could be confusing. Creating an instance of the CEmailServer class in this way might indicate that we are also creating a new e-mail server. Something that might be more expressive would be to use a static method that creates an instance representing an existing mail server:
CEmailServer theServer =
CEmailServer.GetServer(theAddress);
The second and third lines of code that are difficult to understand have the same problem—they use Boolean values as method parameters:
theMailbox.Open("stevencl", true);
theMailbox.Send (theMsg, true);
In these code snippets, how are we supposed to know the effect of passing true to both methods? There's really no way other than looking up the documentation with the effect that someone reading this code, say in a code review, could easily make the wrong assumption about what the code will do. One easy fix, of course, is to create a well-named Boolean variable, assign it the value true, and pass it in to the method instead of the literal Boolean value. One drawback to this solution, though, is that someone reading this code then has to trace back through the code to find the last point at which the variable is assigned before calling the method, to know what value is passed in. In some cases, another solution might be to use an enum to represent the set of possible options. For example, CMailbox.Open might take an AuthenticationOption enum exposing NTAuthentication and UsernamePassword values. In that case, the call would change to:
theMailbox.Open("stevencl",
AuthenticationOption.NTAuthentication);
Using the cognitive dimensions this way, problems related to progressive evaluation, work-step unit, and role expressiveness were uncovered (note that there are other issues with this API, some that would be uncovered by considering other dimensions and others that would be uncovered by considering other scenarios). Some of the problems are easy to fix without a deep understanding of the developers using the API, such as the role expressiveness issues. The progressive evaluation and work-step unit issues, however, require a better understanding of what users expect from an API before being able to design a solution that meets a user's needs. Knowing what your users expect from an API provides you with a direction in which to chart the design of the API.
What Will Users Want?
In most cases, it's not always easy to figure out the right fix for a potential API usability problem. You may be able to come up with a solution, but it's difficult to know whether or not the fix will work for the developers that the API is designed for. This is why understanding your users is so important.
We use personas to represent our understanding of who our users are. In particular, the Visual Studio Usability group has defined three main developer profiles or personae that describe the stereotypical behavior of three main developer groups:
- Opportunistic.
- Pragmatic.
- Systematic.
For each profile, we have defined what developers matching that profile would expect from an API, in terms of the cognitive dimensions framework. Thus, we now have a way to judge whether an API is usable by comparing the API evaluation with the developer profile requirements. What's more, we are able to display such comparisons in graph form, as in Figure 1.
Figure 1: Cognitive dimensions analysis comparing an API analysis with a developer profile.
The black dotted line in the graph represents the API being evaluated, in this case the fictional e-mail API. The spokes in the "wheel" represent each of the different dimensions (spokes are numbered corresponding to the list of dimensions presented earlier. For example, work-step unit, progressive evaluation, and role expressiveness are spoke numbers 4, 5, and 11, respectively). End points (the middle and outer edges) on each spoke represent the end points on a scale for each dimension. Thus, the point on each spoke where the black dotted line crosses it corresponds to the point on the scale for that dimension that the API has been valued at.
The blue line on the graph represents a particular developer profile. For example, it shows that this persona prefers APIs that have small work-step units (the inner edge on the work-step unit scale), support progressive evaluation at the line of code level (the inner edge on the progressive evaluation scale), and have rich role expressiveness (the outer edge on the role expressiveness scale). If you compare the points where the blue line crosses each spoke with the point where the black line crosses each spoke, you get an idea of how well the API matches the persona's ideal API. Hopefully, you can see that the e-mail API doesn't really hit the mark for this persona (there are big differences between the points where both lines cross on most, if not all, of the spokes in the graph).
Cognitive dimensions have really helped us get to the root cause of issues that we have observed in the usability labs. For example, in one study, we observed lots of developers spending a large amount of time in the help docs looking for code samples that would show them how to accomplish a given task. The first interpretation of this data was simply "Fix the help docs!" However, when we used the cognitive dimensions framework to describe the issues, it became clear that the reason the developers weren't successful when they were searching through the help was because what they were looking for simply didn't exist. The API they were working with exposed a set of abstractions that were at the wrong level for these particular developers. They expected a particular type of abstraction to be exposed by the API but since it wasn't, they couldn't find anything about it in the help docs. As a result, the API team redesigned this API to expose abstractions more in line with what developers were expecting. When we retested the API, it worked much better.
At the very least, the cognitive dimensions framework provides developers with a common vocabulary with which to talk about and discuss API usability. The benefits of such a common vocabulary are twofold:
- First, the language shapes the distinctions that are considered important to attend to. By explicitly describing each of the dimensions, it is more likely that they will be attended to.
- Secondly, the language lets developers discuss issues using terminology that they can assume will be understood by others who are also familiar with the framework.
Conclusion
To ensure that you know how your customers expect your API to work, designing the API through scenario-based design is a starting point. You can also use the cognitive dimensions framework to describe how well the API meets your customers' requirements and to gather feedback from customers. You don't need expensive usability labs to be able to do good usability studies. All you really need are time, patience, willing participants, and a framework with which to understand the results of your analysis.
Steven is a usability engineer at Microsoft, working on Visual C++, the .NET Frameworks, and the WinFX libraries. Prior to joining Microsoft, he was a developer at Motorola, building development tools for Smartcard applications. Steven can be contacted at [email protected].
