Best practices for fixing actionable verbatims

To find the verbatims for the articles you own, use one of the following methods:

  • For one single article use the SkyRay Chrome add-in. It will provide you the verbatims for the last 90 days on a given article.
  • Use the SkyEye Search to search on the alias of an author, product, or service, which provides a summary containing a section with all verbatims.
  • The dedicated Verbatims PowerBI report also allows filtering by author, product, service and shows the respective verbatims.

SkyEye uses an ML model to categorize the verbatims and help authors cut thru the noise to easily find the most actionable ones. See below in this article for the descriptions of those categories. In all reports the ML-determined category is shown next to the verbatim. Also international verbatims are machine translated to EN. The accuracy of the ML model for categorizing verbatims is ~85%. The report on Verbatims PowerBI also has links to overwrite those categories.

What customers expect when they leave a feedback

Understanding customers is key to building great content and helping drive adoption and customer satisfaction. Because of scale, this understanding will always be limited – but there is a lot we can do to improve the "signal" with data in ways that are clear and actionable. There are a number of ways to get a better understanding of customers and one of the many channels where customer voice is extracted is through customer feedback verbatim.

Customers’ biggest expectation in the feedback loop is that we address the issues expressed in the verbatim. Feedback and comments can also be directly actionable for content owners, localization teams, and site engineering teams and we need to respond to feedback, act on it and let customers know what’s been done to make their experience better based on the data. If not, customers will cease to give feedback.

Few examples of the broad categories of customer feedback verbatims include:

Request Example
Request for more information - Please explain like you would to a kid what Microsoft Authenticator does!(Multi-Factor Authentication)
- Any tips on how to do repository authentication? Can't seem to find any docs on it. (Service Fabric)
- How many IPs one can add to one NIC? How many NICs one can add to a machine? (Virtual Network)
Requests for specific fixes - you should update the correct installation link to the sap NetWeaver library.(PowerBI)
Requests for specific examples - Would be nice to have an example demonstrating interop for custom struct types and methods for a library e.g. gmp, gtk etc. (.NET Core)
International experience - WHY I SEE SQL COMMANDS TRANSLATED IN ITALIAN??? (SQL Server)
- Please display Microsoft sites in English when I navigate to English sites, don't magically transform them to Dutch without asking for it.(Azure Active Directory)

Impact of verbatim fixes and responsiveness on CSAT

Analysis show that addressing an actionable verbatim on a timely manner has a direct correlation with article CSAT value. The more customer feedback verbatim we address on a timely manner, the highest the article satisfaction rate will be. The following two figures show the impact of addressing actionable verbatim on customer satisfaction. The green shade is the actual CSAT value of the article while the yellow shade depicts the predicted CSAT. If all the actionable verbatim of the article is addressed, then there can be anywhere between 2%-47.8% increase in CSAT depending on the count of actionable verbatim of the article during the week in target.

Example of an article weekly CSAT plot with (Yellow) and without (Green) addressing actionable verbatim

Example of an article weekly CSAT plot with (Yellow) and without (Green) addressing actionable verbatim.

Verbatim categories

Our content get more than 1M customer feedback verbatims a year and it needs high resources to read and address all these verbatims. Instead, verbatims are being auto grouped into 22 categories of issue types using a machine learning text classification tool. This tool auto categorizes each and every customer feedback verbatim into one of the predefined categories. The definition of each verbatim categories is listed below:

Issue Category Name Description
Form or Presentation Presentation-StyleVisuals The user is commenting about the look of the website.
Presentation-BrokenLink The topic contains broken links.
Site-CannotFind User cannot find what they are looking for.
Site-Feature Comment about the website.
Site-FeedbackControl Comment about the feedback control on the website.
Content Content-Accuracy The article content is inaccurate.
Content-BadExample The example is not working or inappropriate.
Content-Incomplete The content is incomplete.
Content-NeedExample The article needs an example.
Localization Loc-BadMT The machine translation is inaccurate.
Loc-HelpExperience The localized content is hard to use.
Loc-NeedsLoc The topic needs to be translated.
Loc-Other Miscellaneous localization issues that do not fit any other category.
Loc-PrefersENU The user prefers English language content.
Loc-Translation The translation is inaccurate.
Product Product-Experience The comment is about the product user experience.
Product-International The comment is about the product from a foreign perspective.
Product-Link A download link is broken.
Product-Sentiment General product feedback.
Product-SubscriptionSupport Feedback about the subscription model or usability.
Kudos Kudos The article was helpful.
Other Other The comment does not fit any of the other categories.
Rant Rant Offensive comment, spam, or other remarks that are not actionable.

Using a bug workflow to keep track of actionable verbatims

For writers who opt-in to receive a weekly auto bug on actionable customer feedback verbatims, you can specify the Service/Product and verbatim category that you want to follow in the auto bug process and email to APEX Data Science Team. Once a given Service/Product is enrolled in the verbatim auto bug process, each writer can filter actionable verbatims by different dimensions such as author, topic, etc …

How can writers override the Machine Learning classified categories?

Classifying customer feedback verbatims into one of the 22 predefined categories is machine learning process and the model currently has an average of 88% accuracy, meaning for every 100 customer feedback verbatims, 12 will be misclassified to other categories. Writers have an option of overriding the current category of a verbatim and re-categorize them into new categories.

We can do this using two options:

SkyEye Homepage

Option 1:

Click Verbatims in the SkyEye home page and then click CategoryFix link of verbatim we want to fix:

Example: The verbatim text “¿que es un FEP?” which is translated as “What is a FEP?” belongs to Content-Incomplete category instead of Rant.

CategoryFix

By clicking the CategoryFix link of this verbatim, we can change from Rant to Content-Incomplete.

TextClassifierTool_1

Option 2:

Click Text Classifier Training tool available in SkyEye home page

TextClassifierTool_2

The main difference between Options I and II is that verbatim fix using option I will be available in the reports within 24 hours while verbatim fixes using the Text Classifier Training will be part of the model re-training process which occurs twice a week.

What to do with verbatims that are ‘actionable’ but not by writers?

Some of the customer feedback verbatims that we get are product related and are not “actionable” by writers. In such cases, currently, we don’t have a process but we encourage writers to channel those actionable product verbatims into their respective product teams.