Common Measurement Mistakes: Volume 1
We all need a methodology. But simply having a methodology does not guarantee success.
What is a methodology? A methodology is often just a system of measurements accompanied by an acronym. Nowhere is it said a methodology must to be accurate, precise or reliable. There are “methodologies” on the market today with margin of error rates in the double digits, but the people using them still have confidence in those methodologies.
A truly useful methodology is accurate, precise, and reliable. Otherwise the methodology is garbage in, garbage out. Inaccurate and imprecise methodologies lead to poor decisions – and to a false sense of confidence in those decisions.
The way we see it, there are 11 common measurement mistakes. We’ve already explored a few of them in previous posts such as Confusing Causation and Correlation and Confusing Measurement with Feedback. Through the next few posts we will talk about the remaining nine.
Common Measurement Mistake: Drawing Conclusions from Incomplete Information
Every day, your business generates a tremendous amount of data – but that data may not tell the full story. Let’s say your analytics show visitors spend a relatively high amount of time on a particular page. Is that page great – or is it problematic? Maybe visitors simply love the content. Or, maybe they are getting stuck due to a problem with the page.
Possibly your call center statistics show average call time has decreased. Is a decrease in average call time good news or bad news? When calls end more quickly, costs go down, but have you actually satisfied callers or left them disgruntled, dissatisfied, and on their way to your competition? Without additional information to help better evaluate the data, you simply cannot know.
Never draw conclusions from any statistical analysis that does not tell the whole story.
Common Measurement Mistake: Failing to Look Forward.
Every company seeks to look forward, and measuring customer satisfaction after an activity or transaction is certainly helpful, but what if you also want to better predict the future? Measuring customer satisfaction by itself will not provide the best view forward. Using a complete satisfaction measurement system – including future behaviors and predictive metrics such as likelihood to return to the site or likelihood to purchase again – generates leading indicators that complement and illuminate lagging indicators.
Common Measurement Mistake: Assuming a Lab is a Reasonable Substitute.
Usability groups and observation panels are certainly useful and have their place; the problem is the sample sizes are small and the testing takes place in a controlled environment. Say you bring people into a lab and tell them what you want them to do. Does that small group of eight participants represent your broader audience? Does measuring and observing them when they do what we tell them to do provide the same results as real users who do what they want to do? Observation is helpful, but applying science to the voice of customer and measuring the customer experience through the lens of customer satisfaction is critical to success.