Common Measurement Mistakes: Volume 4
Alas, the final volume of our “Common Measurement Mistakes” series – where we examine common mistakes people make when measuring the the customer experience – is here. This is a long but good one and we hope it helps you understand why measuring by proxy can be detrimental to business decision making.
If you missed the first three installments of this series, you can read volume 1 here, volume 2 here, and volume 3 here. You should also look at past posts, Confusing Causation and Correlation and Confusing Measurement with Feedback, which are also common mistakes when measuring the customer experience.
Common Measurement Mistake: Measurement by Proxy
Trying to measure customer satisfaction by measuring a behavior like task completion is commonly referred to as “measurement by proxy.” When measuring by proxy there may at times be a correlation between task completion and satisfaction, but all too often that is not the case.
The key is to identify causation. How many times have you completed a particular task… but still left dissatisfied and vowing never to do business with the company again?
The same phenomenon occurs if you attempt to measure customer loyalty by evaluating customer recommendations or by the likelihood customers will make a recommendation. Either way the end result is measurement by proxy; you attempt to determine one attitude or intention by measuring another. Doing so can create significant measurement noise and render your measurements useless in the process.
To highlight the point, let’s look more closely at task completion: some measurement tools measure task completion as a proxy for measuring customer satisfaction. The underlying theory assumes that when a customer completes a task the customer must therefore be satisfied.
That theory falls apart if a software update takes ten minutes to download and another twenty minutes of struggle and frustration to install. The customer may have completed the task but is far from satisfied and may never return. Worse, the customer may say negative things to others and generate negative word of mouth.
Here’s another example. Say you visit a store to find a special tie for a party you will attend tonight. You find the tie, but locating an employee to ring up your purchase is a challenge. After ten minutes of searching you find a salesperson. He is less than friendly, bordering on rude. Do you still buy the tie? Yes, because you don’t have time to go elsewhere… but as you leave the store you vow never to return.
At the party you receive a nice compliment on the tie (your shopping experience was awful but your sartorial judgment is impeccable) and you tell the story about the terrible service you received, compounding the impact of your bad experience by generating negative word of mouth.
Task completion only measures – no surprise – whether a task was completed. Task completion does not measure satisfaction and does not measure future intentions. Task completion is a poor stand-in for customer satisfaction, but since task completion is data that can be gathered fairly easily many businesses and even experts yield to temptation and use task completion as a proxy for customer satisfaction.
Using proxies is easy. Measuring well is hard. The result of using proxies is measurement and management by inference rather than management based on real data, real intelligence, and real knowledge.
Take the practice of measuring loyalty based on recommendations. McDonald’s customers are satisfied – otherwise McDonald’s would not enjoy its current market share – but McDonald’s customers also do not tend to be particularly vocal. People who love Big Macs may not be particularly likely to recommend Big Macs to others.
In large part that tendency is due to the nature of the product and to the way people wish to be perceived. (In-N-Out Burger customers, on the other hand, tend to be much more vocal about their love of the franchise.)
Some people may be likely to recommend Whole Foods, especially if they wish to be perceived as health- and environmentally-conscious. On the other hand, some may not be as likely to recommend the Wal-Mart grocery department to their friends even though they are incredibly loyal Wal-Mart customers. A number of products – deodorants, toilet paper, dandruff shampoos, etc. – fall into this “recommendation paradigm.” We tend not to share our dirty laundry or our less-than-flattering secrets.
Personality can also play a major role in whether we recommend products or services. Many highly loyal customers simply do not recommend products or companies to others.
Perception also can greatly influence whether you are likely to recommend, but perception does not have the same impact on customer loyalty. Say you find great clothing at a discount retailer. Some may recommend the discount retailer to others, but many will not because they prefer that others assume they buy their clothing from high-end retailers. The influence of perception does not impact their level of loyalty but does impact their likelihood to recommend.
Finding ways to get customers to recommend your business, and measuring their likelihood to recommend your business, is smart business and often generates substantial revenue. But never use recommendation as a proxy for satisfaction or loyalty. Never use satisfaction as a proxy for recommendation or for loyalty.
And while it should go without saying, never use loyalty as a proxy for satisfaction or recommendation.
If you decide to measure recommendations then by all means measure recommendations – but never try to infer loyalty in the process. The relationship between recommendations and loyalty is not causal, even if at times the relationship does show correlation.
Research from universities and corporations around the world consistently proves that satisfaction is causal and is a key driver of recommendations and of customer loyalty.
Remember, a truly useful measurement methodology is accurate, precise, and reliable. Otherwise it is just garbage in, garbage out. Inaccurate and imprecise methodologies lead to poor decisions – and to a false sense of confidence in those decisions.
So remember to measure right, manage forward and make a difference in your company.