Measuring the Benefits of Structured Authoring at BMC Software
In April 2004, the Information Development organization at BMC began a pilot project to explore the benefits of structured authoring. We believed structured authoring would benefit our organization, but we needed to pilot the technology and processes to validate our assumptions and strengthen our business case.
We divided the pilot into three areas: content, process, and metrics. The content team analyzed existing content and designed a new information architecture and supporting technology. The process team analyzed information development processes and recommended changes to accommodate structured authoring, while the metrics team defined metrics to measure progress against the project’s objectives. This article focuses on the metrics area.
Why Metrics Matter
Metrics are measures that we can use to determine the costs, assess the effects, and communicate the impact of a project. Metrics are a fundamental part of doing business. Metrics enable teams to clarify what they plan to do and later communicate the value of what they did.
To implement structured authoring at BMC, we needed to be able to communicate the benefits using real numbers. For example, when communicating with upper management, we needed to show that investing in this project would provide healthy financial returns and improve customer satisfaction. When talking to writers, we needed to show that structured authoring would not only improve their deliverables, but also benefit the writers directly in their daily work.
Identification of Metrics
The metrics team was tasked with defining metrics to measure how well we achieved the project’s objectives, which we defined as follows:
- Improve information quality
- Improve information development efficiency
We began by stating desired outcomes that supported the objectives. We then refined the outcomes to reflect what was feasible to test during the pilot phase of the project. By focusing on specific outcomes for the pilot, we eliminated duplicate or invalid metrics. (In some cases, the same metrics could support multiple outcomes.) The remainder of this article describes some of the metrics that we developed.
Metrics for improving information quality
We determined that structured authoring would enable us to improve the quality of information in two primary ways:
- Improved information consistency
- Improved information usability
Improved information consistency
The metrics in Table 1 measure topics that are written differently. These metrics measure the percentage of increased consistency after switching to structured authoring.
|Increase the number of topics that are written consistenly, thereby increasing consistency by n%.||Number of topics that
||Seven writers write about the same subject. Four of them write the topic differently, and three write it the same way. Thus, 57% of the topics are written differently, and 43% are written the same way.
When structured authoring is used, the percentage of topics that are written the same should increase.
|Increase the number of topics that are structured consistenly from book to book, thereby increasing consistency by n%.||Number of topics that
||Writers A and B each write a section about configuring a product. Writer A primarily uses tables and paragraphs, while Writer B uses specific headings and numbered steps.
Across 10 topics in the section in Books A and B, six topics are structured differently, and four are structured the same way. Thus, 60% of the books are structured differently, and 40% are structured the same way.
When structured authoring is used, the percentage of topics that are structred the same way should increase.
Table 1. Metrics for Improving Information Consistency
Improved information usability The metric in Table 2 measures the amount of time to locate a topic in a book. This metric determines the percentage by which improved organization, consistency, and accuracy increase usability in product documents.
|Decrease the amount of time it takes to locate a topic, thereby increasing usability by n%.||Amount of time to locate a topic.||
||Currently, it takes the average user three minutes to locate a specific configuration procedure.
When structured authoring is used, the amount of time it takes to locate the information should decrease.
Table 2. Metric for Improving Information Usability
Metrics for improving information development efficiency
We determined that structured authoring would improve information-development efficiency in the following ways:
- reduced duplication of information
- reduced development time
- reduced translation costs
Reduced duplication of information
The metric in Table 3 measures the number of topics that were reused in one or more books. This metric determines the percentages of increased content reuse and decreased volume of information.
|Increase the number of topics that are reused, thereby:
||Number of topics that are reused in more than one book.||
||Four writers each write 10 unique topics that have similar information but are written slightly differently. When structured authoring is used, one writer writes the 10 topics. The other writers reuse the topics without modification.
Reuse of information is increased by 25%. The number of topics in the repository is decreased by 75%. (Instead of storing 40 topics, the repository now stores 10.)
Table 3. Metric for Reducting Duplication
Reduced information development time
The metrics in Table 4 measure the amount of time required to format content and rewrite topics. These metrics determine the percentages of reductions in formatting time and duplication of effort. The metrics also show how automation reduces the amount of time it takes to reassemble information.
|Decrease the amount of time spent formatting content, thereby decreasing information development time by n%.||Amount of time to format content
Formatting includes applying Adobe FrameMaker templates, tagging topics, managing page breaks, and rendering PDF files.
|Estimate the amount of time spent formatting tasks.||Writers spend approximately 20% of their time formatting or laying out pages for deliverables.
When structured authoring is used, much of the formatting is automated, thus decreasing the amount of time spent formatting information.
|Decrease the amount of time spent reworking topics, thereby decreasing information development time by n%.||Amount of time spent reworking topics
Reworking topics includes reformatting content, cutting and pasting content, or modifying reusable content.
||Four writers coordinate the use of 10 common (shared) topics in each of their deliverables, but each writer slightly modifies the shared topics.
The first writer spends three hours writing each topic; each of the other writers spends two hours per topic, for a total of 90 hours.
When structured authoring is used, the first writer still writes the 10 topics for a total of 30 hours. The other writers reuse the topics without modification. It now takes each of the three writers a half hour to locate each topic in the database and import it into their books (compared to the previous two hours spent reworking each topic). Total time for all writers is decreased from 90 hours to 45 hours, thus decreasing development time by 50%.
|Be able to reassemble common components into different deliverables or output types, thereby decreasing information development effort by n%.||Amount of time to assemble a book that contains topics that are included in other books||Determine how long it takes to manually obtain content from various books to include in one book.||Writer A is creating Book A for a product suite. Book A will contain topics that currently exist in Books B, C, and D.
Reassembling the information to create the book can take days, if not weeks.
With an automated system, reassembly time is reduced to minutes, or even seconds.
Table 4. Metrics for Reducting Information Development Time
Reduced translation costs
Content reuse can significantly reduce the cost of translating information. The metric in Table 5 measures the percentage of reduced duplication, thus reducing translation costs.
|Increase reuse, thereby decreasing the number of topics that need to be translated by n%.||Number of topics that are translated more than once||
||Books A, B, and C contain 40 topics, 10 of which are similar but slightly different. The translator spends two hours per topic. Translating the 10 topics for each of the three books takes 60 hours.
When structured authoring is used, the 10 topics are written to be reused.
Translating the 10 topics only once decreases translation costs by 33%.
Table 5. Metric for Reducing Translation Costs
Baselines and ongoing metrics
At the beginning of the project, we realized the importance of using metrics to establish an initial baseline measurement. A baseline is a snapshot of the metrics at a given point. The initial baseline represents the starting point, using your current infrastructure and writing methodology.
To determine whether our metrics remain valid, we will reevaluate them and take additional baselines periodically, comparing each baseline to the previous one. Doing so can validate our assumptions and ensure that we are meeting project objectives. We will change or eliminate metrics or define new ones as needed.
Metrics are critical to measuring the success (or failure) of a project. While the metrics described in this article measure quality and efficiency, they also provide a foundation for calculating the cost savings of structured authoring. For example, the efficiency metrics indicate hours saved. Based on the hourly cost of a writer’s or editor’s time, we can calculate total cost savings. Similarly, with the translation and reuse metrics, we can calculate reduced translation costs based on reduced word counts.
The sample metrics in this article provide actual measurement definitions and calculation formulas. Using this data, you can build anything from an executive-level business case to a detailed usability analysis of content.
About the Authors
Lead Information Developer
Carla Jennings is a Lead Information Developer at BMC Software, Austin. She holds a Bachelor of Business Administration in Computer Information Systems from Texas State University (formerly Southwest Texas State University). She has worked in the software industry since 1989 as an information systems auditor, technical writer, and technical writing manager. Carla is a senior member of STC.
Information Development Manager
Wendy Shepperd is an Information Development Manager at BMC Software. She holds a Bachelor of Business Administration in Management Information Systems from the University of Texas. She has worked in the enterprise software industry since 1992 as a developer, technical writer, trainer, project manager, and manager with leading companies such as BMC, Compaq, HP, and Trilogy. She is currently managing a project to implement structured authoring using DITA. Wendy is a member of CIDM, OASIS DITA TC, STC, and Content Management Pros.