"Even the genius asks questions."
- 2Pac, Me Against The World
One of the most controversial topics we come across in our discussions with Net Promoter practitioners is the issue of the two-question survey. Most everybody has an opinion on the topic. However, the facts are somewhat hard to find. I recently visited a couple of Net Promoter customers who have invested some serious time into understanding which survey tactics are working across a group of their peer companies. Based on what I have seen in the market, their findings are not far from the mark.
Let's start by defining our terms. When we talk about two-question surveys, we are typically referring to a "recommend" question on a zero-to-10 scale, along with an open text question along the lines of "What is the one thing Company X could do to improve." It is tempting to discard historical approaches to survey development in favor of this dramatically simplified approach.
In most cases, the rationale behind such a move can be summed up by one of the following:
Many surveys are so long that they guarantee low response rates from a very selective group: folks with nothing better to do than fill in lengthy surveys
High response rates (greater than 60%) are highly desirable. One proven way to achieve these rates is to dramatically reduce the number of questions.
Analyzing what drives people to answer the way they do (commonly called "driver analysis") by using follow-up questions is an imprecise science that can often be misleading.
Following up on every response is the most valuable element of a CEM program. It does not require lengthy data capture or lots of analytics.
It's cheaper to ask fewer questions if you are engaging in telephone surveys.
If you are hoping for a simple, definitive answer to this question, you will be disappointed. At the risk of wading neck deep into the issue, let's see if we can make some sense of all of this by considering the evidence we have gathered from the field.
Let's start with the easy one: cost. If the largest cost of your CEM program is telephone data capture, then you have bigger problems to solve. Typically the only companies still capturing data via telephone are those firms that lack electronic relationships with their customers. The cost of collecting data by phone often has more to do with incomplete records than it does with time spent on calls. Once someone is on the phone, it costs very little to add an extra question. However, this discussion is moot when you consider that many companies make major decisions -- and spend millions of dollars -- based on the data generated by these programs.
Suffice to say, the real argument doesn't concern cost. It's all about effectiveness and best practices. Many business people believe that surveys are onerous and too long to complete. It only makes sense that a two-question survey would dramatically improve response rates and represent a lower burden on respondents. Shorter surveys drive higher response rates.
The debate, however, should not be around whether shorter is better, but what the tradeoff curve looks like. If 30 questions is too long and 40 is ridiculous, is 10 about right? What about going from 10 to 2? What benefits can we expect?
To find the answer to these questions, we evaluated a large number of surveys currently being used by Satmetrix customers. If you include transactional surveys, we're talking about millions of complete surveys per month. The response rates vary from single digits to greater than 60 percent. The problem is that the range of response rates does not nicely correlate to survey length.
While it is generally true that very long surveys often have very poor response rates, best-in-class results are not restricted to very short surveys. In fact, some companies with 10 to 20 questions obtain better than 50 percent response rates, while companies with much shorter surveys receive poor response rates. Other factors are at work, factors that seem to have at least as significant of an impact on response rates as do very short surveys.
Of course, none of this is of any relevance if there is nothing to be gained by asking additional questions. In some cases, additional questions buy very little. Surveys that have been designed by committee look ominously like an Act of Congress, with so many earmarks from each department that the final design is bloated. If appeasing the various interested parties is all that is accomplished by adding questions, we should really just drop the lot and stick to the Ultimate Question after all.
However, I do believe driver analysis, if done properly, is of value. It's one thing to say that a company has an NPS of 40 percent. It is quite another to determine how that company has achieved that score. In our experience, an open second question creates an interesting histogram of topics, but it does not really explain their significance or correlate them to loyalty. Few senior managers are interested in a program that doesn't answer the "why" or verify the assumptions of the management team. And, unfortunately, an open-ended second question doesn't always tell us why customers recommend certain products and services.
Much of the criticism of driver analysis is well deserved. Low response rates, poor respondent selection, poor management and weak analysis have turned a useful tool into a poor indicator for many firms. But don't blame the tool -- blame the use of the tool. Correctly implemented, there is no reason why you can't obtain good driver analysis from a short survey.
I'll leave the final word on this issue to a customer who recently visited half a dozen firms that have implemented two-question surveys. His conclusion was that these companies represented two distinct types of programs. In one type, the surveyors assumed that they already knew the answer to the "why" of their NPS, and they seemed confident that their management would not challenge their conclusions. The other type had done driver analysis in the past, and therefore assumed that the drivers would not change over time.
"But what is the point of the program if it has no impact on the drivers?" this customer asked me.
What indeed?
