Measuring the Benefits of Structured Authoring at BMC Software

Home/Publications/Best Practices Newsletter/2006 – Best Practices Newsletter/Measuring the Benefits of Structured Authoring at BMC Software

CIDM

April 2006


Measuring the Benefits of Structured Authoring at BMC Software


CIDMIconNewsletter Carla Jennings and Wendy Shepperd, BMC Software

In April 2004, the Information Development organization at BMC began a pilot project to explore the benefits of structured authoring. We believed structured authoring would benefit our organization, but we needed to pilot the technology and processes to validate our assumptions and strengthen our business case.

We divided the pilot into three areas: content, process, and metrics. The content team analyzed existing content and designed a new information architecture and supporting technology. The process team analyzed information development processes and recommended changes to accommodate structured authoring, while the metrics team defined metrics to measure progress against the project’s objectives. This article focuses on the metrics area.

Why Metrics Matter

Metrics are measures that we can use to determine the costs, assess the effects, and communicate the impact of a project. Metrics are a fundamental part of doing business. Metrics enable teams to clarify what they plan to do and later communicate the value of what they did.

To implement structured authoring at BMC, we needed to be able to communicate the benefits using real numbers. For example, when communicating with upper management, we needed to show that investing in this project would provide healthy financial returns and improve customer satisfaction. When talking to writers, we needed to show that structured authoring would not only improve their deliverables, but also benefit the writers directly in their daily work.

Identification of Metrics

The metrics team was tasked with defining metrics to measure how well we achieved the project’s objectives, which we defined as follows:

  • Improve information quality
  • Improve information development efficiency

We began by stating desired outcomes that supported the objectives. We then refined the outcomes to reflect what was feasible to test during the pilot phase of the project. By focusing on specific outcomes for the pilot, we eliminated duplicate or invalid metrics. (In some cases, the same metrics could support multiple outcomes.) The remainder of this article describes some of the metrics that we developed.

Metrics for improving information quality
We determined that structured authoring would enable us to improve the quality of information in two primary ways:

  • Improved information consistency
  • Improved information usability

Improved information consistency
The metrics in Table 1 measure topics that are written differently. These metrics measure the percentage of increased consistency after switching to structured authoring.

Desired Outcome Metric Measurement Example
Increase the number of topics that are written consistenly, thereby increasing consistency by n%. Number of topics that

  • appear in multiple books
  • provide the same information in each book
  • are written differently
  1. Count the total number of similar topics that appear in multiple books.
  2. Count how many of the topics are written differently, and how many are written the same way.
  3. Calculate the percentage of topics that are written differently.
Seven writers write about the same subject. Four of them write the topic differently, and three write it the same way. Thus, 57% of the topics are written differently, and 43% are written the same way.

When structured authoring is used, the percentage of topics that are written the same should increase.

Increase the number of topics that are structured consistenly from book to book, thereby increasing consistency by n%. Number of topics that

  • appear in multiple books
  • provide the same information in each book
  • are structured differently
  1. Count the total number of similar topics that appear in multiple books.
  2. Count how many of the topics are structured differently, and how many are structured the same way.
  3. Calculate the percentage of topics that are structured differently.
Writers A and B each write a section about configuring a product. Writer A primarily uses tables and paragraphs, while Writer B uses specific headings and numbered steps.

Across 10 topics in the section in Books A and B, six topics are structured differently, and four are structured the same way. Thus, 60% of the books are structured differently, and 40% are structured the same way.

When structured authoring is used, the percentage of topics that are structred the same way should increase.

Table 1. Metrics for Improving Information Consistency

Improved information usability The metric in Table 2 measures the amount of time to locate a topic in a book. This metric determines the percentage by which improved organization, consistency, and accuracy increase usability in product documents.

Desired Outcome Metric Measurement Example
Decrease the amount of time it takes to locate a topic, thereby increasing usability by n%. Amount of time to locate a topic.
  1. Conduct a usability test by asking users to locate specific content.
  2. Determine how long it takes users to locate the content.
Currently, it takes the average user three minutes to locate a specific configuration procedure.

When structured authoring is used, the amount of time it takes to locate the information should decrease.


Table 2. Metric for Improving Information Usability

Metrics for improving information development efficiency
We determined that structured authoring would improve information-development efficiency in the following ways:

  • reduced duplication of information
  • reduced development time
  • reduced translation costs

Reduced duplication of information
The metric in Table 3 measures the number of topics that were reused in one or more books. This metric determines the percentages of increased content reuse and decreased volume of information.

Desired Outcome Metric Measurement Example
Increase the number of topics that are reused, thereby:

  • increasing the reuse of content by n%
  • decreasing the volume of content that is stored in the repository by n%
Number of topics that are reused in more than one book.
  1. Count the total number of topics.
  2. Count the number of topics that are reused.
Four writers each write 10 unique topics that have similar information but are written slightly differently. When structured authoring is used, one writer writes the 10 topics. The other writers reuse the topics without modification.

Reuse of information is increased by 25%. The number of topics in the repository is decreased by 75%. (Instead of storing 40 topics, the repository now stores 10.)

Table 3. Metric for Reducting Duplication

Reduced information development time
The metrics in Table 4 measure the amount of time required to format content and rewrite topics. These metrics determine the percentages of reductions in formatting time and duplication of effort. The metrics also show how automation reduces the amount of time it takes to reassemble information.

Desired Outcome Metric Measurement Example
Decrease the amount of time spent formatting content, thereby decreasing information development time by n%. Amount of time to format content

Formatting includes applying Adobe FrameMaker templates, tagging topics, managing page breaks, and rendering PDF files.

Estimate the amount of time spent formatting tasks. Writers spend approximately 20% of their time formatting or laying out pages for deliverables.

When structured authoring is used, much of the formatting is automated, thus decreasing the amount of time spent formatting information.

Decrease the amount of time spent reworking topics, thereby decreasing information development time by n%. Amount of time spent reworking topics

Reworking topics includes reformatting content, cutting and pasting content, or modifying reusable content.

  1. Count the number of topics that are now reused.
  2. Count the number of hours spent reworking the information.
  3. Calculate the reduction in effort due to reuse.
Four writers coordinate the use of 10 common (shared) topics in each of their deliverables, but each writer slightly modifies the shared topics.

The first writer spends three hours writing each topic; each of the other writers spends two hours per topic, for a total of 90 hours.

When structured authoring is used, the first writer still writes the 10 topics for a total of 30 hours. The other writers reuse the topics without modification. It now takes each of the three writers a half hour to locate each topic in the database and import it into their books (compared to the previous two hours spent reworking each topic). Total time for all writers is decreased from 90 hours to 45 hours, thus decreasing development time by 50%.

Be able to reassemble common components into different deliverables or output types, thereby decreasing information development effort by n%. Amount of time to assemble a book that contains topics that are included in other books Determine how long it takes to manually obtain content from various books to include in one book. Writer A is creating Book A for a product suite. Book A will contain topics that currently exist in Books B, C, and D.

Reassembling the information to create the book can take days, if not weeks.

With an automated system, reassembly time is reduced to minutes, or even seconds.

Table 4. Metrics for Reducting Information Development Time

Reduced translation costs
Content reuse can significantly reduce the cost of translating information. The metric in Table 5 measures the percentage of reduced duplication, thus reducing translation costs.

Desired Outcome Metric Measurement Example
Increase reuse, thereby decreasing the number of topics that need to be translated by n%. Number of topics that are translated more than once
  1. Count the total number of topics.
  2. Count the number of topics that are translated more than once.
  3. Determine how long it takes to translate a topic.
  4. Calculate the savings for translating a topic only once.
Books A, B, and C contain 40 topics, 10 of which are similar but slightly different. The translator spends two hours per topic. Translating the 10 topics for each of the three books takes 60 hours.

When structured authoring is used, the 10 topics are written to be reused.

Translating the 10 topics only once decreases translation costs by 33%.

Table 5. Metric for Reducing Translation Costs

Baselines and ongoing metrics
At the beginning of the project, we realized the importance of using metrics to establish an initial baseline measurement. A baseline is a snapshot of the metrics at a given point. The initial baseline represents the starting point, using your current infrastructure and writing methodology.

To determine whether our metrics remain valid, we will reevaluate them and take additional baselines periodically, comparing each baseline to the previous one. Doing so can validate our assumptions and ensure that we are meeting project objectives. We will change or eliminate metrics or define new ones as needed.

Summary

Metrics are critical to measuring the success (or failure) of a project. While the metrics described in this article measure quality and efficiency, they also provide a foundation for calculating the cost savings of structured authoring. For example, the efficiency metrics indicate hours saved. Based on the hourly cost of a writer’s or editor’s time, we can calculate total cost savings. Similarly, with the translation and reuse metrics, we can calculate reduced translation costs based on reduced word counts.

The sample metrics in this article provide actual measurement definitions and calculation formulas. Using this data, you can build anything from an executive-level business case to a detailed usability analysis of content. CIDMIconNewsletter

About the Authors

carla - BW

Carla Jennings
Lead Information Developer
BMC Software
carla_jennings@bmc.com

Carla Jennings is a Lead Information Developer at BMC Software, Austin. She holds a Bachelor of Business Administration in Computer Information Systems from Texas State University (formerly Southwest Texas State University). She has worked in the software industry since 1989 as an information systems auditor, technical writer, and technical writing manager. Carla is a senior member of STC.

Wendy-BW

Wendy Shepperd
Information Development Manager
BMC Software
wendy_shepperd@bmc.com

Wendy Shepperd is an Information Development Manager at BMC Software. She holds a Bachelor of Business Administration in Management Information Systems from the University of Texas. She has worked in the enterprise software industry since 1992 as a developer, technical writer, trainer, project manager, and manager with leading companies such as BMC, Compaq, HP, and Trilogy. She is currently managing a project to implement structured authoring using DITA. Wendy is a  member of CIDM, OASIS DITA TC, STC, and Content Management Pros.

We use cookies to monitor the traffic on this web site in order to provide the best experience possible. By continuing to use this site you are consenting to this practice. | Close