From Symptoms to Solutions Using Survey Data Analysis
Using Surveys at Point of Use and Applying their Results for Improvement and Design
Metrics are a powerful source of evaluating the success of projects, particularly if the metrics are derived from user input. Carefully crafted surveys can elicit key data from users that tell us what is important to their way of doing business. From this information, we can develop solutions and measure the success of those solutions.
In Tech Pubs organizations, we often make assumptions about our users in the context of how we use information, not necessarily (or consistently) within our users’ frame of reference or expectations for usability. As a result of a recent project to resolve product issues, we learned that the more we can incorporate feedback from our users at work in situ (for instance, at point of use) the better chance we have at making their jobs easier and improving the user experience and performance of the product. Surveys on how users access and use our information can be an effective method for collecting meaningful information to which we can respond with proposed improvements and develop metrics to measure that improvement.
If constructed carefully, surveys also can be a clever mechanism for designing tools to meet the needs of our users. Based on recent benchmarking we did in the San Francisco Bay Area, companies are now seeing the combined value of consolidating information in a single venue and the power of collaboration where critical issues are discussed and resolved in real time and globally. New technology offers us the ability to capture before-unknown information and transform that information into sustaining documentation. The challenge is how to design information delivery to accommodate that collaboration. Surveys to collect design- and information-retrieval preferences of your users can be used for key metrics of success.
Surveys also can lead to innovation. Solutions derived from user-survey data analysis can be game-changers if we think creatively and pay attention to what the data is telling us. We learned from surveys that our field engineers have a wealth of tribal knowledge but had no effective means to share it with their co-workers scattered around the globe. Our field service engineers were already suffering from “email fatigue” which became overused and, over time, was ignored. Because technology now allows us to repurpose data in almost infinite ways, we quickly realized that we found the problem for a pre-conceived solution: online collaboration. Raw or “source” data from users can be very powerful, especially if we begin to recognize it as its own “genre,” a genre that I call “socialized information.” The source data from collaboration venues (online discussion threads, for instance) created by technical users can not only be a controlled, effective means for knowledge transfer, but it’s a goldmine of information that may not otherwise be discovered or available to us.
From this recent survey project, we gained some keen insight about new opportunities:
- 1. conducting in-situ surveys to determine root causes of problem symptoms
- 2. applying solutions with portals and technical collaboration
- 3. leveraging a new “genre” of information from the collaboration space
- 4. extracting metrics from survey results to know if the solutions work
Our Story—the Symptom, Problem, and Solutions
The insight I refer to originates from an investigation we did, teamed up with Tech Support and Tech Training. Our customers were experiencing “long down events” where our products were inoperable because of some performance problem and inoperable for an unacceptable length of time. Product downtime was symptomatic of some unknown variables for which we needed to find the root cause.
We conducted extensive user surveys of our field service engineers (FSEs) and learned there was no silver bullet, but rather, an array and combination of issues. Tightly integrated within these issues was one problem which we could (possibly) do something about. Among many other causes, the field survey data told us:
- ♦ It was difficult to find documentation needed to perform required tasks
- ♦ Field Engineers had to go to too many places to find the information they need
- ♦ There was too much redundant–at times inconsistent–information
Bottom line: We learned that our FSEs had too many places to look to find information, that it was scattered across several different databases, and that this added to the service repair time. Here was one area we believed we could make a positive impact to the company.
We decided to create a “1-Stop Shopping” environment to reduce the places our users had to go for information, consolidate some of that information within a single venue, and then provide the means for technical collaboration worldwide. The solutions we pursued were
- 1. Field Portal—consolidate information
- 2. Technical Forum—provide real-time exchange, capture field knowledge
- 3. Metrics—use variations of “time” as a measure to determine if we were successful
A strategy evolved during this effort that underlines a basic premise of how technical communicators may need to start thinking about their content. In a development workflow, our process evolved into the sequence shown in Figure 1.
Figure 1. Developing a 1-Stop Shopping Workflow
As we probed into root-cause analysis and possible solutions to this situation, we recognized something very obvious. Online, interactive, and collaborative are all adjectives that differentiate and characterize content apart from what we developed just a decade ago. As such, we need to define, create, manage, and distribute information using different paradigms than we did a decade ago. Are manuals, even if electronic and online, still sufficient? We said no. Information is too dynamic and increasingly complex to rely on static words on a (web) page.
Problem in a Snapshot: The Data Silos
Over the years, as our products became more sophisticated and complex, information to support products of this integrated design of hardware and software became stored in several different databases, depending on the type of information or its purpose. All of these databases contained critical information needed by FSEs, but they all operated independently, redundantly, and none “talked” to each other (see Figure 2).
Figure 2: Data Silos
Business Case for “1-Stop Shopping”
This silo’d data resided in separate repositories, creating excessive and redundant content on FSE laptops with procedures and service information isolated from related information. An emerging idea was that Engineering, Manufacturing Test, Field Service, and Tech Support organizations needed a forum for communication that
- ♦ was available from one informational hub where FSEs can access information they need to do their job
- ♦ provided simplified information retrieval
- ♦ eliminated data redundancy (we had 4x redundancy in some databases)
- ♦ reduced the volume of information needed to support different work scenarios
Solution #1: Field Portal—Collect and Consolidate
For our first proposed solution, the Field Portal, we wanted to develop a field service community environment to bring FSEs to one place they could go to find information: our vision of 1-Stop Shopping. Our goal was to reduce the time it takes for an FSE to locate the data or tool they need and provide a jump off point for all related information. FSEs needed a centralized search feature to look at all content, placing results from multiple sets of documentation next to each other. In addition, FSEs needed a method to collaborate on new information, which is often the most pertinent to the technical solution. A positive side effect we hoped for was speeding up the transfer of knowledge from the Engineering group to the FSE, especially when Engineering has new and important information, by streamlining information access from engineering to the field. Ideally, providing 1-Stop Shopping reduces the ambiguity about where the information can be found and is made available from multiple sources. It was our intent that the Field Portal would provide this capability.
Vision: Tool and Information Integration
Rather than have the FSE launch multiple tools to complete a service call, we set out to integrate information, process, and tools within the Field Portal for every service call, from initially diagnosing the problem to customer notification that the product is back up (from start to finish). Links to complete the entire service call from the Field Portal were provided, even when the system to perform that step resided elsewhere (see Figure 3).
Figure 3. Tool and Information Integration
How Did We Design for 1-Stop Shopping?
1-Stop Shopping sounds like an interesting objective to meet, but how do you get there? The original survey was successful in telling us where the source of some problems were, but we needed follow-up surveys to find out how FSEs use information and design solutions. Again, survey results gave us a wealth of information on where to focus our efforts to meet the majority of user preferences. (Note: The most valuable data was turning survey results into meaningful metrics for improvement. More on this later.)
Data analysis on these next-round survey results proved to be invaluable for understanding how to go after key problems to find key solutions. One follow-up survey was to obtain Field input on the design and features of the Field Portal. This survey had 19 questions, with 72 responses (out of 120). And voila! Literally, there it was–how they used information, why it was a problem using it or finding it, and more importantly, their preferences for how they needed information provided to them to be more successful in their work (see Figure 4).
Figure 4: Sample Data Analysis on User Preferences
Key Design Input for the Portal
From this survey input, we learned more about the needs, constraints, and behavior preferences of users for locating information. In addition to the summary data above, we learned, more specifically, that our FSEs
- ♦ prefer PDF format
- ♦ rely on a variety of search features
- ♦ want information from one location
- ♦ do not like multiple windows opening
- ♦ want links to related Field Replaceable Units (FRUs)
- ♦ want links to related troubleshooting guides (TSGs)
And lots more! In the one example below, we received a mixed response to a design question on preferences for access to information. The responses gave us a design strategy for web page elements, links, layout, and navigation. We learned that a robust search feature is critical, and the more ways you can provide searching mechanisms, the more likely you are to meet the variety of preferences your users have for how finding information. We now provide all these different methods for retrieving information based on the statistics in Figure 5.
Figure 5: Sample Survey Results on User Preferences
Solution #2: Technical Forum—Collaborate, Capture, and Transform
The evolution of social media has leveraged its way into the technical corporate arena. Companies are learning that “socialized information” has value, but that it needs to be captured then structured. A collaboration site provides a method to
- ♦ communicate an issue to multiple people when it happens
- ♦ allow experts to respond and exchange probable cause
- ♦ enable global (situational) irrelevancy
- ♦ collaborate toward a solution
- ♦ capture that exchange for others to use
Why the Technical Forum as a Solution?
Other than email, there was no efficient method for technical collaboration from cross-functional experts. Nor was there a way to quickly capture and repurpose that information into sustaining documentation and knowledge. As a result, we historically experienced
- ♦ incidents occurring before headquarters was aware or could respond
- ♦ regions relying on (and growing) their isolated pools of tribal knowledge
- ♦ delay in disseminating solutions across regions
- ♦ solutions not efficiently disseminated among technical organizations within the company
- ♦ knowledge loss
- ♦ missed opportunities for best practices
Because our FSEs encounter unique issues with our installed products, they need expert help in resolving those issues. A communication channel was needed that was
- ♦ cross-functional
- ♦ bi-directional
- ♦ structured
- ♦ unstructured (discussion)
How Did We Know the Technical Forum Would Work?
We are often asked how we knew a collaboration space would work. The truth? We didn’t. (And with only a partial implementation at time of this writing, metric data is still premature, and results are not yet conclusive.) Our field survey asked “would a wiki (collaboration site) be useful to you?” 48.6 percent replied Yes and 40.3 percent replied Not Sure. We saw this as a potential of 88.9 percent in favor, provided the “Not Sure” could understand how it would work and would find it useful. As is required with most innovation, we took the risk on this potential! Several sub-sites were developed off the Field Portal as communication channels among different field groups.
From its most basic design, the Technical Forum was developed as a bi-directional, multi-user venue for
- ♦ online discussion
- ♦ communication between technical participants
- ♦ solving performance issues and capturing results
- ♦ disseminating that information for broader use and increased product knowledge
To this end, we also wanted to positively impact some corporate initiatives, such as reducing an industry metric of “mean time to repair” (MTTR). Most service companies are measured by this criterion and spend a lot of effort in product design, service strategies, and training to keep MTTR below industry averages. A “differentiator” for our company is having low MTTR for our products. We believed that if the Forum was implemented successfully, Tech Pubs could contribute directly to this differentiating metric.
Another objective of the Forum is to standardize field communication protocol. Field communication to and from Headquarters had no standard for content, format, distribution, or even tools (emails, calls, escalation processes). With a Forum, we can centralize a real-time venue for communication that doesn’t rely on emails or late night phone calls.
The Forum is also a tool to expedite solutions to the Field. By narrowing the choices of where to get critical information, this venue can be used to push out critical and timely information worldwide, controlled from one source.
Participation and Governance—Panel of Experts
A collaboration space, such as our Forum, had to have some clear rules of behavior to be an efficient tool and not get out of control. We solicited VP sponsorship to gain authority and support. We also targeted cross-functional participation since technical issues are never solved in a vacuum. We invited the pertinent stakeholders to participate:
- ♦ Technical Support
- ♦ Field Service Engineers
- ♦ Engineering
- ♦ Guest Subject Matter Experts (SMEs) as necessary
The idea emerged that the forum should be governed by an appointed Panel of Experts to keep discussion in control, focus on finding root causes, and always driving to a solution. Forming the panel (which is still in process) does require negotiation with the field service corps to have technical experts available in each global region 24/7 as panel members: a “follow the sun” strategy with one expert in every region during their regional daylight hours. We have tried to avoid bureaucracy with this governance where we can, allowing any participant to initiate a discussion thread and not limiting the number of active discussion threads.
The huge benefit we saw with this process was data capture. Data capture occurs when the content of a discussion thread (technical issues, for instance) leads to a solution, the discussion is closed, the information is captured, transformed into structured content, and repurposed into output/deliverables as appropriate. To us, this process produces a new genre of socialized technical information that in its own right has a high value.
Additional responsibility of the Panel is to provide guidance on how content is to be dispositioned. Figure 6 depicts the flow of information originating from technical discussions to solutions, then captured and repurposed into sustaining forms of content. This information flow is an example of what is now often referred to as “curating” information: information disseminated for its best use, purpose, and format.
Figure 6: Data Capture Process from Collaboration Forum
Solution #3: Develop Metrics to Know if Solutions Worked—The Metrics of Time
Because the symptom of one of the problems was the time it took to find information, we created metrics from our surveys to find the source of the company’s pain: escalations due to poor product performance. Time as a metric can be an effective way to look at how you contribute–or detract–from the company value. We focused on three metrics of time that were relevant to the FSEs, were related to chronic problems in the field, and produced data we could quantify:
- ♦ Time it takes to find what you need
- ♦ Time from escalation to resolution
- ♦ Cycle time for transforming unstructured knowledge
Metric #1: “Time to Find It”
We followed up with our users to learn how long it takes our FSEs to find information they need. I can warn you, be careful what you ask for … the survey came back with cringing results, as shown in Figure 7.
Figure 7: Sample Survey Feedback on Time to Find Information
Based on field feedback, we learned that the time it takes to find information impacts
- ♦ total service time
- ♦ accuracy of the repair
- ♦ confidence of the FSE
- ♦ confidence of the customer
If we could reduce the time to find information, the product would be repaired faster and the customer happier. Based on the above results, our management gave us a new metric to reduce information-finding to <five minutes! Again, be careful what you ask for. But, we now have a specific goal with methods to achieve it.
Metric #2 : Measuring the Effectiveness of Transformed Knowledge
With a collaboration space, we can collect, capture, and transform timely and important information. But how effective is capturing real-time technical exchange and repurposing it? Can this captured information make a difference with the customer and our field corps? How would we know? We took a look at these questions and realized that we can know because we can quantify what we can control:
- ♦ The information products that result from technical collaboration
- ♦ The cycle time from collaboration to resolution to a sustaining information product
- ♦ How quickly this new, critical, and timely information is disseminated to global sites
This information, still being collected at time of writing, should give us insight into the extended value of the Technical Forum as a tool for real time technical problem solving–both immediate, and in a sustaining form.
Metric #3: From Escalation to Resolution
By measuring the time from an escalated down event to the time it was resolved can further add to the value of how effective (or not) a Technical Forum could be. We can quantify how long it took to resolve the problem (MTTR) and compare to traditional escalation methods using email. Our success metric is to determine if the new Forum shortens the time from report of problem to resolution. Results are in progress, and we anxiously await validation of our premise or learn that we need to go back to the drawing board for better solutions.
Technology brings new opportunities for technical communication professionals to add value to the bottom line. We can use surveys to determine root causes of problem symptoms. Using new collaborative tools can leverage the social aspect of technical information, which can be captured and responded to in real time. We can then apply solutions with portals and technical collaboration. Our opportunity is to repurpose information for broader use, influencing better outcomes using a new “genre” of information from the collaboration space.
Transforming the technical exchange into sustaining documentation can be invaluable to the field corps, while extracting metrics from survey results can let you know if your solutions work. Using surveys to collect the “as is” condition
- ♦ gives data to support the need and the design of collaborative tools
- ♦ provides a baseline from which to measure success
Chona Shumate has been a senior manager of the technical publications group at Cymer, Inc., a semiconductor company in San Diego, CA for the past 15 years. Prior to that, she worked as a technical writer for 8 years in Silicon Valley. She has a background in usability testing and topic-based/single-source information design. Among her recent projects, she implemented an XML/DITA/CMS integrated system for developing, publishing, and repurposing large volumes of product information, as well as a solutions-based Field Portal for Cymer’s field service personnel. She is very active in establishing documentation and training standards for the semiconductor industry. She holds a masters degree in Technical Communication and is a member of STC, CIDM, and the International SEMI Standards – Metrics Committee of North America.