Using a Balanced Scorecard to Measure the Productivity and Value of Technical Documentation Organizations
Measuring success in technical documentation groups has always been a challenge. Many companies use traditional measures of productivity such as pages per day, books published, or schedules met. These measurements are remnants of a manufacturing economy, based on factory output and inventory. For developers of knowledge content, these metrics do not provide an accurate view of their productivity.
Knowledge work is a growing component of the US labor force. A large percentage of the US labor force is made up of knowledge workers, including technical documentation professionals. As the world moves toward a knowledge-based economy, technical documentation groups need to move beyond the traditional productivity measures with an additional focus on quality, customer satisfaction, defect rates, timeliness, and costs. These measurements can be quantified and compared using the balanced scorecard approach.
The Balanced Scorecard Approach
The balanced scorecard was introduced by Dr. Robert Kaplan, a professor at the Harvard Business School, and Dr. David Norton, the president of a consulting firm, in a Harvard Business Review article in 1992. Kaplan and Norton described a system of measuring key drivers in the business and showing how they interrelate. Their article described how just measuring output or financial performance will not show the progress of an organization; a variety of topics should be measured and compared.
By looking at a mixture of elements, companies can balance quality and costs. The balanced scorecard is a management tool that can be used by department leads to see how their groups are performing. It is also a measurement tool; metrics can show the overall health of the organization and any changes in productivity.
Knowledge work cannot be measured accurately using traditional metrics for output and costs because the process of developing this content is largely invisible. Knowledge work is not like an assembly line process, where the amount of time required to bolt in a widget is measured. Different metrics are needed to measure success. Management will always be interested in measuring how their employees are performing. However, a new set of metrics is needed for this complex type of work.
The balanced scorecard offers a more even-handed approach than simple financial metrics. It introduces a tapestry of measurements that lead to a more well-rounded picture of the organization. The scorecard is a tool that can help measure knowledge worker productivity. It is also a compact and clear way to communicate the value of content throughout the enterprise.
The scorecard should align with the business strategy. Each company, and even organizations within a company, might need to track different issues. What is important to one group is not necessarily important to other groups. Even in technical documentation, there are differences. If a group is trying to shrink the size of its documents, maybe words per topic is a useful metric. For another group, the concern might be writer-to-developer ratios. While it is important to have several different categories so that the metrics are balanced, the detail of what is measured depends on what is important to that organization.
A Broader Definition of Productivity
The balanced scorecard provides a more comprehensive definition of productivity than a simple relationship between input and output provides, especially for knowledge work. Rather than using a few output-based measurements, a balanced scorecard uses a broader range of inputs for comparison.
The balanced scorecard provides a view of the organization from four perspectives: financial, customer, internal, and learning. The organization can develop metrics, collect data, and conduct analysis relative to each of these perspectives. These main perspectives were defined by Kaplan and Norton, but the metrics used to evaluate each perspective depend on the priorities of the organization being measured. Productivity becomes not just a number but a complex system of inputs.
Each scorecard perspective consists of a set of metrics. Each metric gets a score, and that metric is weighted by how important it is to the organization. The sum of the set of metrics for each category is tallied, providing a measurement for each perspective.
In the end, management often wants one number to measure a complex system. The balanced scorecard can show how each of the four perspectives is doing. The four scores can be combined to produce an overall productivity score for the organization.
Four Categories of Measurement
Kaplan and Norton define four main categories of measurement, based on the four perspectives of the balanced scorecard (Figure 1).
Figure 1: Four Categories of Measurement
These categories apply to technical documentation groups in the following ways:
- Financial – This category shows how the group delivers content to meet organizational and budgetary objectives. Measurements might include cost per topic or feature, cost per defect fix, or other measurements based on output.
- Customer – This category demonstrates the focus on customer satisfaction by responding to customer feedback. Actual customer satisfaction numbers could be tracked, as well as number of defects and average time to fix a defect.
- Internal – This category shows timely delivery of quality content. Internal metrics might also track the number of support calls related to documentation, or comparisons of quality between the product and the documentation.
- Learning – This category shows how the department is increasing skills that benefit their contributions to the larger organization. Metrics might include certifications for product training as well as project management skills.
Steps to implementing a balanced scorecard include the following:
1) Develop a baseline scorecard using established and easily calculated metrics for the four categories. Many of these metrics are probably already being tracked in one form or other. Developing the baseline includes looking at the various metrics that are important to the organization and sorting them into the appropriate categories.
2) Define metrics and goals that reflect company targets, industry averages, or departmental averages, depending on topic. Each metric has a goal. Some of these goals are set by upper management, such as customer satisfaction or cost targets. For metrics that do not have corporate goals, a measurement can be tracked over time and compared to the departmental average. In other cases, comparing the department’s progress against industry averages might yield a useful number. The basis for the goals might vary for each topic within a category, determined by each department’s priorities and history (Figure 2).
Figure 2: Balanced Scorecard—Goals
3) Specify metrics that allow the departmental objectives to be evaluated. Equalize by scoring against a goal and weighting the importance of each area. Each metric is given a score depending on how the department performed on its goal. If a scale of 1 to 5 is used, the goal is usually in the middle (3), with 1 being a disappointing result and 5 exceeding expectations. This score is then multiplied by a weighting, depending on the importance of the metric. Each category gets a value based on the sum of the weighted metrics within it. Then, the scorecard gets an overall score from the sum of the categories (Figure 3).
Figure 3: Balanced Scorecard—Metrics
4) Compile calculations for a significant enough period to be able to create a true baseline scorecard for later comparison. A scorecard done one time is of no value. This method will show its value only after scores are calculated for an extended period. It might be useful to look back at any historical metrics when first creating a scorecard so that a meaningful baseline can be created.
Balanced Scorecard Example
A technical publications group within a software development organization wants to develop a balanced scorecard to show how its group is doing. The group has determined which metrics are important for reporting and has gathered all the necessary historical data.
First, the goals are listed on the scorecard. The metrics to be tracked are listed under the most appropriate category, and the card is ready to be filled in with the details.
Next, the actual measurement is entered for each metric, and this measurement is compared against the goal. A rating is given to the metric depending on how the goal and the actual result compare.
The final step is to provide the weighting for each metric. In this example, each category is balanced, representing 25 percent of the whole. Within each category, the weight must add up to 25 percent. With the weighting added, the scores for each metric can be calculated. The scores for each category are generated, and the scorecard as a whole is given a number for comparison with later measurements.
A Balanced Look
As with any set of measurements, the scores will only be as good as the data. If the data collection is consistent, trends should appear on the scorecard. Each metric, each category, and the scorecard as a whole should be tracked over time for the most effective results.
For technical documentation groups, information content is the product. There must be an effective way of measuring how effectively information content is produced, and the balanced scorecard provides a well-rounded model for measuring productivity. The scorecard approach provides visibility into several perspectives of a department’s efforts.
Cisco Systems, Inc.
Neville Fleet is the Senior Documentation Manager for the Knowledge Management and Delivery (KMD) group at Cisco Systems, Inc., in San Jose, California. This group of 90 technical writers has overall responsibility for the majority of the software documentation for Cisco’s Internet Operating System (IOS). Neville has 25 years of experience in technical documentation and product management for major international corporations in the IT industry. Prior to joining Cisco in 2001, Neville held management roles at IBM’s Application Development Group, Mercury One-2-One, and Digital Equipment Corporation. KMD is currently in transition to an XML-based structured content environment.
Cisco Systems, Inc.
Joan Lasselle is founder and president of Lasselle-Ramsay, Inc., a professional services company that develops business information and learning solutions that drive superior user experience, productivity and change. Lasselle-Ramsay focuses on four practice areas: content management, technical documentation, training development and on-the-job information tools. Since 1982, Lasselle-Ramsay has worked with major high-tech manufacturers including Cisco Systems to develop technical documentation solutions for commercial products.
Cisco Systems, Inc.
Paul Zimmerman is a Program Manager in the Knowledge Management and Delivery (KMD) group at Cisco Systems, Inc., in San Jose, California. He counts involvement with departmental metrics and Web 2.0 implementation among his interests at Cisco. Paul has been working in technical communications for over 18 years, including a management position at Lucent Technologies and individual contributions at Octel Communications, Touch Communications, and Comtech Services.
JoAnn T. Hackos
Managing Your Documentation Projects
1994, New York, NY
John Wiley & Sons
Robert Kaplan and David Norton
“The Balanced Scorecard—Measures That Drive Performance”
Harvard Business Review
Wim Van Grenbergen
“The Balanced Scorecard and IT Governance”
Information Systems Control Journal
Thinking for a Living
2005, Boston, MA
Harvard Business School Press