The Illusive, Writing Productivity Metric: Making unit cost a competitive advantage
“I write until I’m finished,” was the curt response to the new doc manager’s query, “When do you think you will be finished with the document?” This was, of course, many years ago, and enlightened writers and managers have come a long way since then. Still, the question of how to effectively measure writing productivity seems as elusive as the Holy Grail.
At the recent CIDM Best Practices Conference in Hampton, Virginia, I hosted a luncheon forum on writer metrics and productivity. Interest was so strong that the restaurant scrambled to assemble a sufficient number of tables to accommodate the demand. Before the first water glass was filled, the conversation started, “How do YOU measure productivity?”
The ensuing discussion was enlightening because it was clear that many managers were (1) dissatisfied with their current measure (if they had one), and (2) most seemed receptive to a more effective unit cost metric. Having migrated to a unit cost metric more than ten years ago, I have found that when calculated properly, it can be fair, valid, reliable, and easy to do.
What is a Unit Cost Metric?
There are several ways to measure cost and output. There are department budget costs, project costs, loaded salary costs, total page counts, and several unit cost measures like cost per page and cost per topic. Budget costs represent the total cost to the company for documentation. Usually the more projects and documents one produces, the higher the cost. But what is the right cost? It’s hard to quantify. Unit cost takes that total cost and divides it by the number of units (projects, features, documents, pages, topics, words) that are produced for that total cost amount.
The advantage of unit cost is that it is scalable, valid, and reliable. If your average cost per document is $1,000 and you produce 10 documents, your total cost is $10,000. If you double your output, you will likely double your total cost. So what budget number is the right number? It depends. It’s just a number that someone higher up wants to make smaller. Even worse, it’s difficult to defend.
The scalability problem with budgets is that some documents are harder to produce than others. Some are longer than others. Some products have more churn than others. There are just too many variables to justify a budget number. What one can’t justify often ends up subject to reduction. What is needed is a measure that normalizes the variables, is scalable, valid, and easy to justify.
Cost per Page
Some years ago progressive managers chose pages as the measurable unit to count. Dividing the department budget by the total number of pages produced yielded the “cost per page” metric that many managers still use today. As was clearly stated at the luncheon forum, that metric does not seem to work for many managers. Why is that?
There is an old adage that says “you get what you measure.” With automobiles, for example, if you measure miles per gallon, you encourage the production of more cars with higher miles per gallon ratings. If you measure horsepower, you will begin to see larger and larger horsepower engines. Over the last decade there has in fact been an increase in both metrics for most automobile manufacturers. You get what you measure.
For writers, if we measure pages, we should expect to see more pages from each writer, and we often do. Year after year, successful writers produce more pages when given that goal. Unfortunately, more pages are NOT necessarily what customers really want. Most customers want more information, but not necessarily more pages. They want fewer pages that are more succinctly written and better organized so they can find and understand what they need to know. They don’t want more pages.
Most writers understand how metrics motivate their work—their yearly rating and salary increases often depend on it. If you want more pages, writers will give you more pages. They will likely be quickly written (no time for editing), loosely constructed (no time for organizing), verbose (more words makes more pages), and difficult to understand (no time to organize one’s thoughts into a logical flow). We need to be careful what we measure because if we measure pages we will likely get lots of them, but customers may not be happy with the result.
Cost per Topic
Some managers have moved away from counting pages to counting topics. So what is a topic? A topic is a self-contained bundle (or chunk) of information that can stand on its own and make sense. When we write on-line help, each on-line topic is a “topic.” When we use information mapping, we are chunking information down to topics, which might be a second or third level section of a chapter. For Unix folks, a “man page” is a topic. For procedural writers, one procedure (“Removing the Oil Filter”) is likely one topic. Once one starts thinking about it, defining a topic becomes a relatively easy task.
By dividing a department’s yearly budget by the number of topics produced during the year we get the unit cost metric of “cost per topic.” At first glance this seems very much like cost per page. How is this metric different? The difference is in the measurement incentive. If we measure pages, we get more pages, which may not necessarily be a good thing. If we measure topics, we get lots of topics, which turn out to be a good thing.
Customers want more topics. The company wants writers who produce more topics per year. By counting topics, we give writers an incentive to make each one smaller and more succinct so they can produce more of them. It encourages writers to use minimalism and get rid of words, sentences, and paragraphs that are unnecessary. It encourages reuse. It encourages process improvement and tool development. The incentive is to get to the core of the information, present it in the simplest possible way, and leverage tools that help make that happen more quickly. Customers like short, succinct topics. Companies like increased output. Writers like to feel more productive. To get more topics out the door, writers will start to think about process cycle time and get rid of process steps that don’t add value. Writers will start looking for time wasters that don’t make the document any more accurate or complete. Counting topics rewards the right behaviors for everyone involved.
There is another benefit to measuring cost per topic. When managers use lower-cost writers in other countries to produce documentation, it’s hard to measure relative productivity. I often hear “are we really saving anything by using writers in varied geographical locations with varied skill sets and experience?” By using cost per topic, it is quite possible that the most expensive salaried writer could actually be the lowest cost writer when measuring “cost per topic.”
Measuring Cost per Topic
The way a manager measures “cost per topic” is critical to the effectiveness of the metric. The first consideration is WHEN to count the topics. For many departments, the best time to count is when the document is completed and published. Each month, the number of touched topics can be counted and plotted on an x-y line graph (topic count on the vertical scale and months on the horizontal scale). Each month would have one total topic count number for the department plotted on the graph.
Unfortunately, most writing departments don’t have sufficient numbers of writers to normalize the distribution of topics that are completed over the course of the year. For example, some months may have huge numbers of topic counts, and other months may have zero. The graph over twelve months looks more like an EKG than a productivity metric, useless, until we make one important, graphical change. If we plot the data on a chart in PowerPoint or Excel, we have the option of adding a trend line. An exponential trend line seems to work the best. With twelve months (or more) of data, the trend line statistically normalizes and smoothes out the data points, and the result is nothing short of amazing.
If your department is continuously improving its cost per topic over time, your trend line should slope down toward the right (lower cost per topic over time). By measuring the delta of the trend line over a twelve-month period, you can calculate the real percentage of improvement in cost per topic for your entire department. When this happens you should tell the world how good you are as a manager. See Figure 1 for more information.
If your trend line slopes up to the right or is level, you are not improving your productivity and need to make adjustments in your department before something bad happens to your team, like a big cut in funding.
The Trend Line is the Key
The trend line is the key to this metric. Here’s why. First, there is usually insufficient data and too many variables between documents and writers. By pooling all the writers’ data into one big department bucket, the individual differences are normalized and the average, represented by the trend line, will likely be fair, valid, and reliable. Like any normal distribution of data points, this one is only true if the type of work performed in the department is fairly consistent over time. That is, the mix of hard documents and easy documents, difficult projects and easy projects, and good writers versus average writers, is about the same from year to year.
Second, writers must buy into the metric. Their first concern will likely be that this is a way to single them out for poor performance. By ONLY using cost per topic as a department measure, and telling writers that you will NEVER use the metric for individual writer comparison, writers will more likely agree to the measurement scheme. Once they see the department improving over time, more will support the measure and look to find ways to make it better. That’s when things really start to improve; when everyone is pulling in the same direction trying to make the number better over time.
Third, the measure is NOT an absolute number (like 30 miles per gallon on an automobile). The metric you want to look at is NOT the data number itself, but the DELTA of the trend line over time. This is a critical point. The goal is continuous improvement, NOT achieving some arbitrary number. Why is this so important? Because even though this measure attempts to normalize productivity, there will likely be insufficient data points to be able to compare scores between teams. Moreover, one doc team with 50 writers may have very easy documents with supportive SMEs and little churn, while another team of 50 writers may have hard documents with uncooperative SMEs and lots of churn. Each team’s yearly cost per topic number will likely be very different, but they can both set a goal to improve their 12-month trend line by 10 percent each year. Nearly every team can improve 10 percent. Percent improvement on the trend line is a valid goal that normalizes the kinds of variations that make absolute metric numbers difficult to compare. It also eliminates the performance fear that “my team’s absolute number might be higher than someone else’s number.” The absolute number is just not that important. The percentage improvement on the trend line, however, is vital to contributing to the productivity improvements that funders like to see.
What is a Touched Topic?
For those keeping score, there is one definition that has been ignored—what is a “touched topic”? When counting topics, I said to count “touched topics.” A touched topic is a topic that has been added or modified in a document. Counting all the published topics is not valid. We need to count only new topics or modified topics, which lead to the question “How do you weight new and modified topics?” The answer is, I don’t. For many departments, the mix of new and modified topics usually stays about the same. As long as there is no drastic change in the mix, there is no need to weight the topics. Why?
Weighting topics has two drawbacks. First, it takes a lot more time. Counting topics is one thing, but to have to make an assessment whether this is a new topic, a 75 percent modified, a 50 percent modified, or a 25 percent changed topic adds way more complexity than is necessary. As long as the mix is relatively constant, it’s not worth the time cost. Second, making an assessment is much harder than just counting. Counting is fast, can be audited, and the results are repeatable. Assessment is messy, takes a lot of time, and gives the assessor an opportunity to tilt the results, even if unintentional. Assessment is a hole in the measurement that is susceptible to gaming the system. It opens up a hole for complaints about unfair assessment. It is difficult to repeat the results in an audit since two different assessors may rate the same topic in different ways.
In most cases weighting just doesn’t matter. When you make stew, a few more carrots, beans, onions, or pieces of meat are virtually undetectable in the final product. If some pieces are large and some are small, the final stew tastes pretty much the same. Stew with only carrots or only beans would be a different matter, but for many departments, the mix of variables is relatively the same, especially when measured across many writers and many projects over the course of a whole year. By keeping the measure simple (counting touched topics) the result is actually more valid, more reliable, and easier to perform.
Years of Data Helps the Trend Line
I recommend keeping years of data on your metric chart. You measure the delta change over a rolling 12-month period, but keep all the data on the chart. The trend line will be more accurate AND you have immediately available historically data to show your funders your improvement over time.
I have been measuring “cost per topic” for about eight years and have found the data to be extremely valid, very reliable, perceived by most involved as fair, and most important of all, beneficial in showing funders and senior management how productive the writing team really is. By including the entire department budget number in the numerator of the calculation there is no expense that is deceptively excluded. It includes salary, benefits, phones, vacation time, holiday time, training time, and goof-off time. By counting every touched topic, writers feel the measure is fair. By showing only the department average, individuals feel protected from personal scrutiny, but will acknowledge that they have a personal stake in the measure. The calculation is simple and takes very little time. No special Excel or PowerPoint skills are required. The data can be audited by anyone if necessary to make sure the numbers are correct. Management loves seeing that productivity improvements cannot be skewed by gaming the system with a complicated formula. Simpler is better.
I strongly encourage every doc manager to try measuring cost per topic. It is a powerful tool that can benefit customers, funders, senior management, and your writing team. If your trend line is going in the wrong direction, you don’t need to show anyone the data. Instead, perform a root cause analysis and find out how you can improve. When your trend line is going in the right direction, tell everyone! Just don’t ask me for my raw number, because comparing my raw numbers with yours is not valid. I could share with you my percent improvement—or, maybe not. It’s a competitive advantage that I don’t want my competitors to see.
About the Author:
Mike Eleder is a Senior Manager at Alcatel-Lucent. He has 32 years’ experience writing and managing the creation of customer documentation for telecommunications and high-tech industry market leaders, and is trained in Six Sigma, CMMI, and Agile development. He has been using cost per topic as a key department metric for ten years.