As you may have already figured out, the “Ultimate Question” that is the foundation of Net Promoter is not actually a question, but questions. Just like Batman cannot exist without Robin or Peanut Butter without Jelly <insert cheesy analogy here>, likelihood to recommend has limited business value if there is not some way of tying context to the response that was given. The numerical values captured from clients is extremely useful for quantitative analysis, including correlation and regression analysis to identify drivers, while the open ended question is extremely useful for qualitative analysis.
Is too much data a bad thing? Not at all if you have the means to process large quantities of data. With numerical data, it is quite easy to ingest volumes of data into a database and crunch numbers with little to no regard for the volume that is being processed. However with unstructured data, like verbatim comments, we face a much more formidable challenge.
As Vince Nowinski, Director of Methodology, Satmetrix, shared during his session today, there are multiple ways of managing qualitative data in the form of customer verbatims.
- Single Punch Categorization – This is where most comment analysis occurs. Verbatim comments are “bucketed” into a single themes or category. Categorization may be more subjective due to interpretation or bias from the person categorizing the comment.
- Multi-Punch Categorization – Similar to the above, however comments are bucketed into multiple themes or categories.
- Comment Intelligence - Using software algorithms, open ended feedback are processed analysis is performed by the incidence of particular words and their relationships with other words within the comment. In contrast to the single-punch and multi-punch categorization techniques which require categories to be created by the person (or people) performing the categorization, comment intelligence uses technology that analyses the patterns which exist due to interreliability of the keywords in a data set. Objectivity is not an issue since software performs the analysis.
The output of comment intelligence allows an organization to uncover relationships amongst different concepts and themes, along with the frequency and likelihood of association of keywords within themes.
For organizations with very large volumes of open ended feedback, comment intelligence solutions may be a good investment simply due to the time and effort required to manually process verbatim feedback.
While verbatim feedback is very useful to dovetail into quantitative analysis, Vince shared the following insights to take into account when using qualitative feedback:
- Qualitative feedback diminishes as surveys lengthen
- Open ended comments require more thought and effort on behalf of the client
- Motivated customers share more and their sentiment influences responses
- Tailored/targeted questions are best for asking verbatim
- Customers are unreliable narrators and raters due to halo effects, fatigue, satisficing, ballistic responding and both cultural and response biases (Is it root cause or top of mind?)
- Don’t’rely just on quantitiative feedback because you’re bound to miss what you don’t measure!
Used wisely, however, verbatim comments can inform thinking about the customer experience, shed light on how customer segments differ and prove better insights into root cause of customer experience issues.
