Measuring Performance in Information Development

Home/Publications/CIDM eNews/Information Management News 08.06/Measuring Performance in Information Development

JoAnn Hackos, PhD
CIDM Director

Once upon a time, an information-development manager in a large telecommunications company discovered that the cost of one project far exceeded the costs of any of the other projects her staff worked on. Investigating further, the manager learned that this project was a victim of poorly defined requirements and specifications, resulting in nearly continuous redirection and rework of the documentation. Nothing was ever written once. Due to continuous changes in direction, the writers spent a large fraction of their project time adding, deleting, and rewriting topics. When the manager reported the findings to senior management, they decided to replace the engineering project manager. The lower productivity among the writers on this project mirrored productivity problems among the engineers.

As information-development managers, we are anxious to increase the productivity of our staff and reduce costs. We’re under great pressure from senior management to measure our efficiency and improve performance. Too often, however, we fall into the trap of measuring the wrong things. We tend to look at the costs of the final results (total project costs) rather than the costs of performance as they occur. Without detailed measurements in place during a project, we have no way to determine which projects have implemented best practices and which are more efficient than others. If we can monitor and measure our best projects, we can perhaps find ways to spread best practices throughout our organizations.

In their article,“Measuring performance in services,” in the February 2006 McKinsey Quarterly (Issue 2), Eric Harmon, Scott Hensel, and Timothy Lukes explain why services are so much more difficult to measure than manufacturing. Service activities vary widely in efficiency and quality because they depend more completely on the experience, skill, and abilities of people rather than machines. And, as we know, people are highly variable. Assign your star performers to a difficult project and you’ll discover that it takes less time to complete than other, seemingly simpler projects. Put an excellent, tested process in place, and projects take less time even though more steps are included and are executed more thoroughly. Take, for example, thorough audience analysis and information planning with a minimalist focus. Every hour spent making better decisions about content can save two or three times as many hours spent producing content that no one wants or needs.

External benchmarks are rarely informative enough

Many information-development managers seek external benchmarks to measure the performance of staff. They ask other managers for the ratios of product to information developers or editors to writers or the number of hours to produce a topic or a page. Harmon et al. point out that external measures are often poorly defined and unreliable. The ratio of writers to developers might be quite low in an organization that produces hardware products with lots of legacy content compared to an organization that is producing new software for a new and challenging market. Asked for the average hourly cost of information developers, one company may quote $60/hour while another quotes half that much. The first company is likely including fully burdened costs over and above salary.

Internal benchmarks may be too high level

Even internal benchmarks may not reveal performance problems. We recognize that some projects are more difficult than others, a recognition inherent in Comtech’s Dependency Calculator (http://www.comtech-serv.com/dependency_calculator.htm). The Dependency Calculator provides a systematic method of calculating the cost of a project depending upon such factors as the complexity of the product, the cooperativeness of the product developers in providing information, the degree to which the product changes during development, and the skills and experience of the information developers assigned.

As the Dependency Calculator suggests, averaging the cost per topic or page across all information-development projects produces numbers that are relatively meaningless and do nothing to help us measure and control variations between the best and the worst projects. To measure performance effectively, we have to identify what makes our projects different from one another and find useful ways to measure the differences.

Aggregate costs hide cost-related details

Even the aggregate costs of developing the information for each project often does not provide the information we need to correct variances in performance. The aggregate costs are buried in all the possible variations that projects naturally face.

  • Some products are more complex and difficult to understand than others.
  • Staff members vary in experience, technical knowledge, and skills.
  • Some projects include more maintenance changes than new writing and vice versa.

    As a result, we find it difficult to make comparisons because the data we report is not the same across projects.

Data collection methods are inadequate

Many managers also have difficulty separating data gathered for financial reporting and data needed to measure and improve performance. Managers frequently report that their data reporting systems allow staff members to report no more than eight hours per day of work and no weekends, despite the fact that they are all working extra hours to meet deadlines. They also know that one staff member includes all the hours devoted to a project, including information gathering, meetings, and thinking time while others include only the time they spend actually composing content.

Develop an internal cost tree

Harmon et al. recommends that managers develop detailed more detailed metrics that help to reveal where costs can be reduced and performance and productivity improved. For each project or project segment, a manager builds an internal cost tree. The cost tree enables you to compare performance across similar aspects of otherwise different projects, identifying ways to reduce costs and learning which activities most affect the overall costs of a project. By recording each project expense and understanding the underlying cause of each expense, you can pinpoint those activities that threaten high performance and cause development costs to increase. You can also pinpoint activities that reduce costs and boost performance, providing you with a set of best practices to introduce to other projects.

Figure — An internal cost tree

Note that in the internal cost tree example, you find that the percentage of topics with no matches in the translation memory is a major contributor to the cost of translation. You discover that one project has a very high percentage of “no” matches even though the project involves previously existing content and has estimated a high percentage of topic reuse. After an investigation, you discover that this project team is rewriting rather than reusing topics. They need to review their reuse strategy and be more careful about avoiding near-duplicate topics.

You also learn that another project has an inordinately high topic-authoring cost with multiple drafts. The total number of topics written during the project is 200% higher than the number of topics in the deliverable, being that many topics have been abandoned. The engineering team has been making many late changes to the product, requiring massive documentation revisions. Combined with the complete absence of product specifications and an uncooperative engineering manager, this project is out of control. In this instance, the cost of poor development practices has been shifted to the information developers.

The first performance problem can be alleviated by educating the staff and enforcing the reuse policy. The second problem points to underlying causes that are another department’s responsibility. Correcting these problems and reducing project costs will take negotiation with the project and product managers.

The key to performance improvements is in the data. The more you know about the cost drivers in individual projects, the more you can ferret out practices that are inefficient and ineffective among your team members and others outside the teams. You can compare project details among all your projects, seeking out examples of best practices that result in reduced costs and better service to customers, both internal and external. With the data in hand, you have an opportunity to build a more cost-effective and efficient operation. By identifying the reasons for variations in costs among projects, you can allocate your staff more appropriately and build on their best practices.

 

We use cookies to monitor the traffic on this web site in order to provide the best experience possible. By continuing to use this site you are consenting to this practice. | Close