Industry Survey: Documentation Excellence Survey Analysis
Documentation excellence, the holy grail of our industry, is an elusive quality. Out of sheer desperation, many managers base their quality assessment solely on a lack of complaints from engineers or other subject-matter experts. These experts often base their judgments strictly on technical accuracy.
In a customer study we performed for a large company, we were asked to investigate the customers’ reactions to the documents’ quality. We found that most users were not able to access the documentation. The interface to the documentation delivered on CD was so poor that users actually had “documentation retrieval experts” whose job it was to look up procedures in the documentation for everyone else in the office.
When the marketing executive received our report, he said, “We know that information is not easy to find, but where are the inaccuracies?” We explained that if users were not able to find the information in the first place, they’re unlikely to find errors. This manager began to realize how complex and intangible excellence is. Although technical accuracy is very important to customers, it is not more important than accessibility, usability, and readability.
Like many others, this manager had too narrow a view of excellence. To broaden our own view and take the pulse of the industry, in 2001 we conducted a survey and a series of interviews under the auspices of the Center for Information-Development Management. The survey was designed to define the criteria for documentation excellence.
Besides the surveys of information-development managers, our information resource included industry experts and others who are often cited as producing excellent documentation. Using the results of the study, we believe it is possible to assess excellence in documentation. However, we believe that the study results show what we have known for many years.
We cannot define excellence solely in terms of standards that look inside the document. Excellence is a measure of customer satisfaction, and customer satisfaction measures ease of access, usability, accuracy, and completeness.
Unfortunately, customer satisfaction is a lagging indicator. That means we don’t know if we’ve delivered a successful product until it’s done. Although we must measure customer satisfaction as the final arbiter of excellence, we have to find leading indicators that tell us if we are on the right track. Leading indicators assess the likelihood that we are following best practices in order to achieve customer satisfaction in the end.
We can certainly concentrate on the standards, defining excellence in terms of good design qualities. These are reliable leading indicators of success, but they are not sufficient. We also must define success in terms of process. By following best practices in our pursuit of excellence, we are able to predict with some success that our customers will be not only satisfied but also delighted.
Study participants agreed that this list of criteria might be used to define excellence in documentation (see the figure below):
- The document is proven to meet customer needs
- The information is accurate
- The document is well-organized and easily accessible
- The writing is concise
- The writing is easy to read
Unfortunately, a simple bulleted list makes it difficult to communicate the inter-relatedness of these qualities. If you were to focus on only one, you would fail to achieve excellence. The confused manager we mentioned above, focusing solely on accuracy, chose to ignore the almost complete lack of accessibility.
Meets customer needs
The central criterion in an assessment of excellence is to meet customer needs. To measure this criterion requires that we assess customer satisfaction after the information is complete and has been delivered. As such, assessing customer satisfaction is a lagging indicator.
Many organizations assess customer satisfaction by a simple yes/no question on a survey. The responses provide us a baseline from which to measure change, but they have little value in themselves because they provide no information. To get the information we need, we must further assess customer satisfaction by studying the success that customers have in using the information to meet their goals.
Customer satisfaction is the one area that can’t be evaluated through competitions and internal studies. It can be partially assessed through usability testing but even testing may not give a complete reading of success.
Unfortunately, assessing customer needs and customer satisfaction is the most neglected criterion for excellence. Again and again, information developers tell us they have no contact with customers.
Customer studies are also important not only to judge customer success with documentation but also to develop innovative strategies for delivering information.
Innovation is possible only if you are truly aware of customer needs and issues. In a customer study we conducted recently, we found that, initially, users had difficulty locating items on the screen because the new operating system was very different from the one on their PC. The best way to communicate about navigation and screen location is through interactive examples.
As a result of the customer studies, we recommended an interactive user guide that permitted customers to see how the new system worked. As innovative as this solution has proven to be, we would never have recommended it without intimate knowledge of user problems. Without that knowledge, it would have been another misuse of technology for technology’s sake.
The second criterion of excellence is information accuracy. Customers tell us, and rightly so, that incorrect information is extremely troublesome. Accuracy, however, is a factor of process. To develop accurate documentation requires a thorough review process and a complete testing program. To accomplish each of these is difficult when the process fails:
- Information is not carefully reviewed by subject-matter experts
- Information is never tested because of lack of time and resources
- Functions change after the documentation has been finished
Too often, products are documented in an interim state that does not accurately reflect how the product is used by the time it is completed. For this reason, it is very important to integrate testing, in addition to technical reviews, into the documentation process and ensure that what you have documented is what your users see.
Another underutilized source of information accuracy and completeness is technical support. Many times, even though they may not be tracking errors in the documentation, technical support personnel can tell you where the most significant technical support issues are concentrated.
Accessible and well-organized
You can have the most accurate documentation, containing everything the users need, but if the information is not easy to find or easy to navigate, you might as well not write it. Users must be able to find information intuitively. Often, information is organized according to the product functionality or design of the menus and dialog boxes in the software rather than in a way users group or think about information. Information architects must be skilled in organizing information, whether on the Web or on paper, for maximum accessibility. They must also be prepared to conduct tests to ensure that real customers can find what they need with minimal effort and in the least possible time.
Concise and relevant
Minimalist principles challenge us to write only what is necessary to support user goals and tasks. Minimalism also emphasizes troubleshooting, explaining items that are known troublemakers. Therefore, to write concisely, with only the words necessary, and to ensure relevance, requires knowing your customer.
With data from customer and usability studies, you can prune information that is not necessary and is not used. Eliminating such information leaves more time to include information about tasks that are difficult. Successful minimalism often means decreased documentation costs as well as increased usability.
Clear and easy to read
When assessing documentation clarity and ease of use, you have no better ally than a good editor well versed in the product and fully aware of the needs of the end-users. Editors can work together with writers to ensure that information flows correctly and makes sense. The only true test of this, of course, is another round of usability testing.
This set of criteria combined defines excellence. If you are to achieve excellence, you must have a three-part process:
- Study your customers and measure their success
- Develop a process that ensures accuracy and good design
- Apply industry best practices for document design
With all three parts in place, you are able to evaluate documentation quality with both leading and lagging indicators.
Just remember the advice of the experts in our field. Excellent documentation is not something people can simply look at and judge definitively. Excellence is determined within the context of where information is used and includes a rigid process of bringing together the end user, subject-matter experts, and technical writers.