Determining the Quality of Your Documentation—A Different Viewpoint

Home/Publications/Best Practices Newsletter/2003 – Best Practices Newsletter/Determining the Quality of Your Documentation—A Different Viewpoint


December 2003

Determining the Quality of Your Documentation—A Different Viewpoint

CIDMIconNewsletter Vesa Purho, Development Manager, Nokia

Quality is not simple thing. There seem to be as many definitions of quality as there are people speaking about it, and the quality of documentation is no exception. Some people think about grammar and spelling when talking about quality, some are more concerned about the usability of the documentation, and yet others rely on good processes to produce good documentation without actually feeling the need to assess the quality of the documents.

I believe in the nececessity of good processes and management practices to help writing groups create good documentation without depending on the skills of any single writer. Using such processes and practices enables a group to create a sustainable output of quality documentation over time and over different projects with different people. During these times of outsourcing, however, you may have to assess the quality of documents without being able to monitor the processes that are used to create them. You just receive the documents, and then you must check their quality and either approve or reject them. Without good, objective quality criteria you may face difficult discussions with a vendor about what actually is good quality and what is not.

A Framework for Establishing Quality Criteria

In this article, I do not intend to present a set of quality criteria, as they vary depending on the type of documentation you create, the type of audience that is reading the documentation, and the context in which the documentation is used. Sometimes the best document is a post-it note stuck on the side of a monitor. Instead, I’m presenting a framework that you can use to establish your own quality criteria.

The framework is similar to an idea presented decades ago by Frederick Herzberg, who studied employee motivation in the 1950’s and 1960’s. In his studies, he learned that the things that make people dissatisfied at work are different from those that increase their motivation. He called the factors that cause dissatisfaction “hygiene factors” and the ones that increase motivation “motivators.” If hygiene factors are not at a sufficient level, employees are dissatisfied with the job. Moreover, even if you increase the hygiene factors past the acceptable level, the employees do not become more motivated to to do the work. Instead, the motivators come into pla, and it is the motiators that lead to the job satisfaction. For more information on the model, see the article “one More Time: How Do You Motivate Employees?” (Harvard Business Review 2003).

Documentation quality criteria also can be divided into two different categories: hygiene and value adding. The hygiene criteria are typically focused on the surface-level factors like content and delivery of the documents, whereas the value-adding criteria are more focused on the users of the documents and their media preferences. Value-adding criteria typically relate to the usability of the documentation.

A Comparison of Quality Criteria—Hygiene versus Value Adding

Failure to meet the hygiene criteria is obvious to users. A failure of hygiene may make the users doubt the credibility of all the information in the documents. For example, if there are a lot of misspellings or the print is of poor quality, the users will notice and might question whether or not the instructions are accurate. On the other hand, increasing the quality of these criteria above a certain level, which depends on the situation, does not really bring any additional value to the users, since they can no longer see the difference. When considering grammar, for example, missing periods can make the text unreadable, but the users may not be able to spot the improper use of commas before “which” clauses. Naturally, if your audience is English teachers, grammar has more importance than if the audience is technicians whose native language is not English. The delivery of the documents is also a hygiene criterion: if the documents do not reach the users, they cannot be used. On the other hand, on cannot make them “more delivered”; they have either been delivered or not.

The value-adding criteria are not so obvious to the users, although users notice if they are missing. Users might think a document is good because the document takes their actual work goals and tasks into account, or the language matches the language they use instead of the language used by the tool designers, or the navigation is intuitive and users can easily find the information they need. The choice of media can also be seen as a value-adding criterion, because, if the documents exist in media appropriate to the context of use, they add more value than if they exist in the wrong media. For example, the trend to push more and more documentation online may actually decrease the value added to the end users, who in many situations could better use a paper document.

When I first thought about this idea, I sent an email message to the Best Practices listserv and to the listserv of the Finnish Technical Communication Society to ask other professionals what they believe are the hygiene and value-adding criteria in documentation. I did not receive many replies, but I would like to thank Tina Hedlund, Leena Haapoha, Petri Pekkarinen, and Sirpa Ihalainen for their input. The following list is probably incomplete, but it gives you some idea of what the hygiene and value-adding criteria might be:

Hygiene Criteria

  • Correct spelling and grammar
  • Correct terminology
  • Working hyperlinks (in online documentation)
  • Overall page design (use of fonts, headings, margins, spacing) so that the text is easy to read
  • Clear graphics (appropriate line weight; use of callouts, colors, and other elements)
  • Accessibility, meaning that the documents are delivered on time through an appropriate delivery channel (mail, Web, with the product)
  • Technical Accuracy
    One might consider that technical accuracy is not a hygiene criterion, but because the documents can be technically very accurate and still not be appropriate to the context in which they are used, I think that it is a hygiene criterion.

Value-Adding Criteria

  • Task orientation so that the documents match the users’ real work goals and tasks
  • Appropriate level of detail taking the different user groups into account
  • Intuitive navigation (table of contents, index, search)
  • Appropriate delivery media for the different user groups and contexts of use
  • Effective use of various presentation elements like tables, graphics, and lists

How, then, does all of this relate to the Herzberg model beyond the fact that I’m using the term “hygiene”? As with the Herzberg model, improving the hygiene criteria does not increase quality beyond the base level; improving the hygiene quality criteria does not add more value for the end users. Therefore, if the users complain about the quality of documentation and all the hygiene criteria are basically in order, you do not increase user satisfaction with the documentation by concentrating even more on the hygiene criteria. Instead, you have to focus on the value adding criteria. On the other hand, if the hygiene criteria are not taken sufficiently into account, it does not help if you increase the quality of the value-adding criteria because the users cannot see past the fact that the hygiene criteria have not been met.

Measurement of Quality Criteria

Hygiene criteria are relatively easy to measure and control. Some of them are met automatically with an overall page design when authoring templates are used. For others, you can set measurable criteria, like X number of wrong terms in Y number of words or X percent delivery accuracy, which can be easily tested. Furthermore, some of the criteria are so basic that one should not even have to measure them (for example, spelling mistakes) but one should be able to rely on the professionalism of the writers. Because these criteria are the foundation on which good documentation is built, you have to have some targets for them, and initially you may have to make regular checks to ensure that the targets are met. However, those measures should in time become obsolete because the results are always good, and you can then use only occasional checks to see that the quality level is kept up.

Setting the targets for the hygiene criteria is not, however, as simple as it first may sound. How do you know how many spelling mistakes or wrong terms a document can have before the users get so annoyed by the mistakes that they don’t want to use the documentation at all? And the targets for the hygiene criteria will not all be the same. For example, the targets set for technical accuracy should be higher than for spelling mistakes. If there is misleading information in the documentation, the users won’t be able to do their tasks or they may even harm themselves, whereas they probably can cope with a few errors in spelling and punctuation. Naturally, sometimes a spelling mistake may actually lead to a technical inaccuracy. As noted before, I’m not setting any specific targets in this article, but I urge you to analyse who the users are and in what context they will use the documentation. Doing so should help you in setting the targets. You could easily say that you require 100 percent accuracy in the hygiene criteria and naturally you may do so, but consider the cost of achieving that goal compared to the benefit you bring to the users. The more you spend your resources on getting the hygiene criteria 100 percent correct, the less time you have for improving the value-adding criteria.

The value-adding criteria are much harder to measure, and it is also difficult to set targets for the. That is one reason why so many organizations rely purely on the hygiene criteria when evaluating the quality of documentation. Because the value-adding criteria can be derived only from careful user analysis and because measuring them often requires the participation of the users, these criteria often remain vague. For example, concerning navigation, a typical target states that “the users can find the information easily.” Now, if you are actually going to determine by examining the documents whether or not the users can find the information easily, you will most probably have many opinions. Unfortunately, the opinion that should count the most, that of the user, is not present. Some of the criteria are such that someone other than the customer can evaluate them. The effective use of tables, graphics, and lists—and here the stress is on the word “effective”—is something that an experienced technical writer can evaluate. Do the elements complement each other? Do the graphics really add value? Can the information appropriately be presented in a table format? And so on.

So here the main question is not, what is the target level? The question is, can we even set some objective targets that can be measured? For some criteria you can (for example, navigation). If in your user analysis you find that users don’t want to, or don’t have time to, search for more than X minutes, then you can set that as a target: Users need to find a particular piece of information within X minutes using whatever navigation method they choose. You can then set up tests with the users during the project to see whether or not you are meeting the target. It is important to evaluate whether or not you meet the target during the project instead of waiting until everything is ready, because changing the whole structure of the documentation late in a project costs too much. If you plan your documentation well, you can do navigation tests using only the architecture and table of contents even when no content is actually ready.

Other criteria are more related to user satisfaction, in which case you cannot observe directly—for example, whether or not the level of detail is appropriate for the users. You have to ask the users to tell you how satisfied they are with the level of detail. In this example, you could set a target for the percentage of the test users that must be satisfied with the level of detail before you think the documentation is good. You could set a target like “90 percent of the test users give a rating of ‘satisfied’ or ‘very satisfied’.”

Quality Criteria and Customer Satisfaction

One question is whether or not you can compensate for the lack of quality of hygiene criteria with the increased quality of value-adding criteria, and vice versa. I think that, as with motivation, to a large extent you cannot. Increasing one’s salary does not make the job any more interesting, and increasing grammatical correctness does not make documentation any more suitable for a task. On the other hand, if the documentation is very intuitive to use so that the users find the information is what they need in the situation they are in, users might forgive a graphic that is a bit blurry or a spelling mistake or two.

In conclusion, the hygiene criteria are the baseline of good documentation and are relatively easy to measure, but they are not enough to really bring value to the users. You have to take the value-adding criteria into account as well. How much you take the value-adding criteria into account depends on the competitive strategies you are using with a product. If you are competing with regard to price, then perhaps it is enough to use just the hygiene criteria as a measure of quality. However, if you are competing with regard to adding value to the customer, you must develop some ways to measure the value adding criteria as well. CIDMIconNewsletter

About the Author