Home/Publications/Best Practices Newsletter/2013 – Best Practices Newsletter/The Balanced Scorecard: Monitoring the Health of Your Organization

CIDM

December 2013

 


The Balanced Scorecard: Monitoring the Health of Your Organization


CIDMIconNewsletter Dawn Stevens, Comtech Services, Inc.

Your team is missing deadlines. Customers are complaining about the content you produce. Your writers are stressed out. Costs of producing content seem to be escalating. You want to adopt new strategies and best practices, but you’re stuck in a rut. Clearly, it doesn’t take an organizational thermometer to see that all’s not as it should be—there’s something affecting the overall health of your organization. But what can you do to diagnose the issues, and more importantly, monitor them in the future so that you can take preventive actions before you face an epidemic of these proportions?

Managers seem to be ever on a quest for the right metrics to determine organizational health. However, an overemphasis on one factor—such as the cost of producing a topic of content—frequently leads to treatment of a symptom in isolation. In effect, it’s as if we measure blood pressure, while ignoring cholesterol levels, and then wonder why our patient had a heart attack. We need to take a holistic approach to managing and monitoring the fitness of our team.

A balanced scorecard enables managers to examine elements of their organizations on a regular basis and helps flag problems before they impact the overall performance of the team. Although each organization may have different tolerances in various areas, use different scales to measure certain factors, or even use different key performance indicators (KPIs) to evaluate an objective, four critical dimensions seem to universally impact the well-being of information development teams:

  • Planning—How well defined are the information products that the organization needs to create?
  • Execution—How well does the organization perform against the plan?
  • Quality—How well does the content produced meet the needs and expectations of the users?
  • Growth—How much effort is put into improving and building core skills within the group?

To gain a complete view of your organization’s ability to meet the expectations of your company and your customers, you must consider all four dimensions. The sections that follow suggest critical objectives within each dimension that will provide a clear picture of the condition of your team and help you focus your efforts on critical skills that can improve your team’s prognosis. Each objective includes possible KPIs that range from simple yes/no answers to longer-term data gathering and analysis.

Planning

A healthy organization takes the time to identify the needs and scope of all its projects. It is deliberate in its actions rather than reactionary. It knows what is currently on the docket and what is on the horizon, and it is staffed to handle the workload.

Objective: Department staffing and budget based on projected workload.
KPIs:
  • Yearly budgets are based on list of well-known and defined projects.
  • Percentage of “ad hoc” projects added after yearly budgets are established.
  • Information development has direct input to each product plan and schedule.

As each planned project begins, additional activities should confirm the allotted schedule and budget before work begins.

Objective: Project budget established before writing begins.
KPIs:
  • Project hours are estimated based on project size and scope as defined in the project plan.
  • Project hours are estimated based on past department performance history.
  • Estimated project hours are adjusted based on project dependencies such as stability, availability of information, and experience of team.
  • Scope is adjusted if the required budget or timeline is not viable or approved.

Note that these measurements do not explicitly indicate what should be used to estimate the hours needed to complete a project. Plenty of debate and guidance exists regarding the appropriate methods—be they hours per topic, page, user story, or entire document. You can choose the measurement that works best for you. What is captured in the balanced scorecard, however, is that you have estimated the time required and that the estimate is rooted in the details of the project and your performance history, not on factors outside of your control, such as random engineering dates.

Similarly, your deliverable schedule should be based on the project scope and required budget.

Objective: Project schedule established before writing begins.
KPIs:
  • Schedule is based on estimated hours and available resources, not just date-driven deadlines.
  • Workload is balanced and prioritized against other projects.
  • Workload is balanced across team members.

The health of your department depends on realistic estimates and schedules. If the scope of the project calls for a larger budget or longer timeline than can be accommodated, scope must be adjusted downward, projects reprioritized, or team members reassigned, rather than crossing your fingers and hoping it works out.

Objective: Appropriate resources assigned and available.
KPIs:
  • Team has skills and experience to fill the roles required by the project plan.
  • Team members are assigned to specific project roles and responsibilities.
  • Team members are allocated specific hours and given deadlines to complete assignments.

Beyond the overall budgeting and scheduling of the project, however, your balanced scorecard should track planning activities that ultimately affect the execution and quality of the end product. In addition to scoping the project to allow enough time and budget, activities must be in place to ensure that the right content is being produced.

Objective: Content plan created.
KPIs:
  • Plan is based on user/task analysis that identifies audience characteristics and information needs.
  • Plan identifies existing topics that can be reused.
  • Plan outlines hierarchy and relationships among topics.
  • Project plan is reviewed and approved by all stakeholders before writing begins.
  • Percent change to project plan is calculated from approval to completion of project.

Execution

A healthy organization meets its obligations according to the plans it has made. In fact, staying on budget and schedule is probably the most common of the dimensions regularly tracked.

Objective: Stay on budget and schedule.
KPIs:
  • Percent of projects delivered on time.
  • Percent of projects delivered within budget.
  • Percent of milestones tracking within budget as planned.
  • Percent of project milestones tracking to schedule as planned.

Clearly, this objective requires regular monitoring, which in turn requires methods for reporting, reviewing, and responding to actual data.

Objective: Monitor progress on regular basis.
KPIs:
  • Team members report actual hours and progress on each assignment weekly.
  • Team reviews actual hours and progress weekly.
  • Team makes adjustments as needed to reverse the trend of all “yellow” KPIs.
  • Team takes immediate action on all “red” KPIs to correct the issue.

Note: Yellow KPIs indicate areas of concern to be investigated while red KPIs indicate areas that are in danger of causing the project to fail.

You might also want to use the balanced scorecard to track some of the critical factors that affect your ability to meet your budgets and schedules. You have likely factored certain assumptions about projects, such as amount of reuse or number of review cycles required, into your estimates. It’s a good idea to use your balanced scorecard to track these assumptions. For example, with a move to topic-based DITA authoring, you may have built a business case on avoiding duplicate work and achieving a certain level of reuse. Your balanced scorecard should track your performance on these goals.

Objective: Maximize reuse.
KPIs:
  • Percent of duplicate topics in repository.
  • Percent of near duplicate topics in repository.
  • Percent of topics referencing content in shared files within the repository.
  • Percent of topics containing greater than 50 percent conditions.
  • Percent of topics with conditional processing at less than sentence level.
  • Ratio of actual reuse percentage to planned reuse percentage.

In addition, there may be other assumptions that are factored into your project histories. However, if those assumptions change, it could have devastating and unexpected impact if you aren’t tracking them on your balanced scorecard. For example, budgets and schedules frequently get waylaid by excessive amounts of rework. The addition of new staff members may temporarily increase the amount of rework as they learn your standards and processes.

Objective: Minimize rework.
KPIs:
  • Percent of assignments that have all required baseline information in place before writing begins.
  • Percent of topics that pass SME review without rework.
  • Percent of topics that pass developmental edit without rework.
  • Percent of topics that pass standards check without rework.
  • Percent of topics meeting the required quality assurance software score without rework.
  • Percent of changed topics that adversely affect other locations where they are reused.

Quality

A healthy organization produces content that its customers need and use. Content is complete and accurate, easily accessible, and understandable to all users. If only efficiency is measured in your department, the quality of the information for end-users may suffer. If you measure only whether deadlines are met or how fast content is created, writers may be rewarded for turning in poorly written content that fails to meet user needs, contains inaccuracies, and violates established best practices and standards. A balanced scorecard must weigh not only the efficiencies, but the resulting quality associated with those efficiencies.

A clear indicator of quality in documentation is whether or not your customers can actually use the documentation to complete their tasks or solve their problems.

Objective: Customers use documentation to successfully complete their task.
KPIs:
  • Customers provide positive feedback on content.
  • Number of support desk calls related to information covered by documentation decreases over time.
  • Support desk encourages use of documentation by showing callers where information is located in the documentation as they answer questions.
  • Trainers encourage the use of documentation during classes.

Note that these KPIs shy away from traditional measurements such as the number of hits or the time spent on a topic. “Hits” are not necessarily “finds,” and spending a lot of time on a topic could just as easily indicate that the content is difficult to decipher than indicating it contains the information users needed. The best measurement of this objective is direct user feedback—surveys, interviews, focus groups, and other customer studies. To effectively track and manage this part of the balanced scorecard, you must have an effective means of gathering user feedback and monitoring user interaction with your content.

Obviously, the corollaries to users being successful in completing a task using your documentation are that the task exists within the documentation, users can find it, it is correct, and it is understandable. By including objectives in these areas, you can shed more light on the reasons that users are, or are not, actually using your documentation. Increasing the scores in these areas should, over time, result in better scores on the first objective in the quality dimension as users learn they can rely on the documentation to have the information they need.

Objective: Documentation includes all user tasks required.
KPIs:
  • Percent of projects with user/task analysis to identify personas, workflows, and tasks.
  • Percent of user goals identified in user/task analysis that are directly supported by the documentation.
  • Percent of support desk calls received that could be answered from the documentation.
Objective: Information can be easily located.
KPIs:
  • Percent of users who find the topic they need on their first try.
  • Percent of topics that appear in the top 10 search results for a search based on their content.
  • Percent of topics in the top 10 results for a search that are relevant to the search text.
  • Ratio of task topics to conceptual and reference topics.
Objective: Content is accurate.
KPIs:
  • Percent of task topics tested against the actual system before publication.
  • Percent of links verified before publication.
  • Percent of topics reviewed by SMEs for accuracy and completeness.
  • Number of publication errors decreases over time.
Objective: Content is understandable.
KPIs:
  • Percent of topics undergoing automated quality reviews that check such things as spelling, grammar, and conformance to standards.
  • Percent of topics undergoing a developmental edit for conformance to minimalist principles.
  • Representative topics are tested by end users for clarity and usefulness prior to publishing.

Growth

A healthy organization does not simply rest on its laurels. It seeks to continually improve and build on its skills and performance. Part of that improvement includes investing in the professional development and growth of the individuals within the organization. The growth dimension of the balanced score card begins with a simple evaluation of whether or not staff members have personal development plans and are measured against them.

Objective: Individual staff members held accountable for personal development.
KPIs:
  • Percent of staff members with a personal development plan.
  • Personal development plans that include efficiency, quality, interpersonal, and education measurements.
  • Progress against personal development plans that is measured regularly (at least yearly) and adjusted accordingly.
  • Reviews, salary adjustments, and bonuses that are tied, at least in part, to the individual’s performance against the personal development plan.
  • Performance reviews that include 360 feedback from superiors, subordinates, and peers.

This objective of the growth dimension is unlikely to change. If performance measurements are in place, they will likely stay in place. The rest of the objectives, however, address the data these measurements provide. For example, do you see improvement and growth from all team members year after year?

Objective: Staff shows improvement over time on all performance metrics.
KPIs:
  • Individual staff members improve the quality of their work over time.
  • Individual staff members improve their efficiency measurements over time.
  • 360 feedback from peers, subordinates, and supervisors improves or remains positive over time.

By including this objective, you measure that individuals, not just the group as a whole, improve their efficiency and quality metrics. Without this insight, you might observe improvements in these areas as a group, without realizing that the improvement comes from a few “star” performers who cover for the rest of the team. The loss of such a star, either through attrition or simple exhaustion, could send the department into a downward spiral, with no early warning. Your balanced scorecard reminds you to make certain that team members contribute equally to the success of the team.

The growth dimension should also reveal when individual skills are stagnant. Perhaps team members are solid employees with consistently good reviews, but their jobs are not evolving over time. They are better and faster at doing something that the industry no longer values or needs. A low score in this objective could be a red flag for your future organization. How will you address advancements and changes in the industry if the team is not aware of them and is not adding new skills?

Objective: Individuals add new skills to their toolkit each year.
KPIs:
  • Department budget is earmarked for staff training.
  • Percent of allocated training budget used within the year.
  • Percent of team taking continuing education classes or attending industry workshops or conferences.
  • Percent of staff actively involved in professional societies within the technical information industry.
  • Percent of staff presenting at conferences and/or writing articles for industry publications.

At an organizational level, growth occurs when innovation and change are part of the culture. It requires more than just putting a change management plan and process in place, but observing whether the process is ever used. Are team members encouraged and eager to innovate or content to do what they’ve always done?

Objective: The culture and environment foster innovation and change.
KPIs:
  • Change management process are in place, used, and understood by team members.
  • Current standards, architecture, processes, and tools are regularly evaluated against industry best practices to ensure they continue to meet the needs of the organization.
  • Number of improvements suggested by team members compared to percent of changes implemented as a result of team member suggestions.
  • Average time required to respond to change suggestions..
  • Average time required to implement a change.

You might also choose to use the growth dimension of the balanced scorecard to track progress on specific corporate or department initiatives. For example, as you move to topic-based development, away from books, you might want to measure department performance in collaboration and teaming skills.

Objective: Staff grows in collaboration and teaming skills.
KPIs:
  • Assignments made by domains, rather than deliverables.
  • Number of deliverables to which each person contributes increases over time.
  • Peers are given the opportunity to review each other’s work and performance.
  • Individuals take ownership in team quality and efficiency goals and are motivated to help each other to meet those goals.
  • Team members are kept well informed about all activities and status on projects through regular emails and status meetings.

In the future, however, as these team skills mature, you might switch your emphasis and tracking to another area, such as content management or writing for reuse.

Forming the Big Picture: Dashboards and Stoplight Charts

Once you have selected the dimensions and objectives you will track, you need to determine how to score and evaluate the measures. In some cases, the KPIs are expressed as percentages that you can easily look up or calculate, but others might require that you define a quantitative score based on a qualitative answer. For example, are yes and no KPIs evaluated at 100 percent for yes and 0 percent for no, or do you need a sliding scale, such as the following?

Objective Score Condition
Individuals take ownership of team quality and efficiency goals and are motivated to help each other meet those goals. 100 Team members proactively offer assistance to others to meet project goals regardless of their own workload.
80 Team members provide assistance when asked by others and they have capacity.
50 Team members provide assistance when asked by management.
0 Team members work in isolation on individual assignments without worrying about overall project goals.

Your balanced scorecard must also reflect the relative importance of those dimensions and objectives. The word “balanced” might imply equal distribution—all four dimensions are equally important to your success. On the other hand, just as a human can function with only one kidney, you might decide that one dimension is somewhat less important than the others. In the following example, growth was weighted much less than the other three dimensions. As a result, its low score, while visible at the dimension level, has much less impact to the overall performance measure. The department still has a healthy score of 90, while clearly needing improvement in growth.

Overall Performance 92.02
Weight
Planning 86.67 30%
Execution 96.00 30%
Quality 100.00 30%
Growth 72.20 10%

Similarly, you can weigh the effect of each objective on the overall dimension score and even the effect of a single KPI on the objective’s score. In the following example, the objective is weighted at 20 percent of the overall dimension, and the first KPI is weighted as twice as important to the objective as the other three. In this case, the weighting of the KPIs results in an overall objective score of 86, and the weighting of the objective results in the 96 percent overall score in the example above.

Objective Score Weight KIP Weight Score
Monitor progress on regular basis. 86 20% Team members report actual hours and progress on each assignment on a weekly basis. 40% 100
Team reviews actual hours and progress on a weekly basis. 20% 50
Team makes adjustments as needed to reverse the trend of all “yellow” KPIs. 20% 80
Team takes immediate action on all “red” KPIs to correct the issue. 20% 100

It’s important to recognize that a “bad” score in a few KPIs, objectives, or even a dimension, does not necessarily mean your organization is suffering. It only highlights areas where you aren’t at your best. Just as you can do a good job with an earache, your organization can continue to perform well even when certain areas aren’t at peak performance. However, left unchecked, other areas may be affected or a critical mass of poorly met objectives can add up to poor performance overall. Use your balanced scorecard to identify those areas that can use some attention and that can be left to run as is.

To flag problem areas, you need to determine the acceptable thresholds within your organization. While some managers expect 90 percent or greater performance measures, you might be happy with only 80 percent. While some might require immediate attention to anything that falls below 75 percent, you might let some decline to 60 percent. Set thresholds on the values in your balanced scorecard to apply a “stoplight” chart interpretation to your scores. In the following example, the manager is not worried about anything with a score of 90 percent of above, is carefully monitoring objectives with scores of 75-90 percent, and has put action plans in place to improve anything with a score less than 75 percent.

[Green]
Project is progressing as planned. greater than or equal to 90%
[Yellow] Project is trending off plan. greater than 75%
[Red] Project is at risk and needs attention to put it back on track. less than 75%

Although it may take some effort to determine the specific objectives and KPIs that are meaningful to your organization and to ensure you are able to evaluate those KPIs, the balanced scorecard is an invaluable tool for managers of information-development groups. It transforms your goals and strategies into an easily interpreted action plan that can help keep your group functioning at its best.

About the Autor:

DawnStevens_ad

Dawn Stevens
Comtech Services, Inc.
dawn.stevens@comtech-serv.com

Dawn Stevens is a Senior Consultant specializing in information development, instructional design, and management consulting. With over 20 years of experience, including 10 years at Comtech, Dawn has practical experience in virtually every role within a documentation and training department, including project management, instructional design, writing, editing, and multimedia programming. With both engineering and technical communication degrees, Dawn combines a solid technical foundation with strong writing and design skills to identify and remove the challenges her clients face in producing usable, technical information and training. Dawn has a keen interest in metrics, editorial and design standards, user and task analysis, and single sourcing strategies. She enjoys helping our clients gain efficiencies through improved publication processes, minimalist writing techniques, and designing for reusability, with an emphasis on XML and DITA solutions.

 [level-members]

We use cookies to monitor the traffic on this web site in order to provide the best experience possible. By continuing to use this site you are consenting to this practice. | Close