Many organizations still rely on asking people what changes they’d like to see in their website or service, neglecting historical research failures like the New Coke or the Aeron chair.
When asking people, you have to be aware that people make confident but false predictions about their future behavior, especially when presented with a new and unfamiliar design. There’s a huge difference between imagining using something and actually using it. In addition, human preferences are rather unstable.
That’s not to say you should quit listening to your customers. But make sure you know what to ask and how to interpret the answers.
Research failures from Malcolm Gladwell’s book Blink:
- The New Coke is one of the most famous research failures. Despite thousands of sip tests and countless efforts to fine-tune the taste based on the customer feedback, the New Coke was a huge disaster. “Gladwell contends that what people say they like in these tests may not reflect what they will actually buy to sit at home and drink over a week or so.” See the full story on Wikipedia.
- The now acclaimed Aeron office chair received very low ratings in early tests. Despite the ratings, the company decided to go on with manufacturing. The rest is history: Aeron became one of the most iconic and best selling chairs in the history of office furniture. And the irony: once the chair became famous, people started rating it much favorably.
- The TV shows “All in the Family” and the “Mary Tyler Moore Show” received poor ratings during pre-air tests. Fortunately, the producers stuck to their ideas and both shows became grand successes.
- In his TED Talk on spaghetti sauces, Malcolm Gladwell argues that the food industry made a big mistake asking people about their preferences and conducting focus groups. Gladwell says that “The mind knows not what the tongue wants. […] If I asked all of you, for example, in this room, what you want in a coffee, you know what you’d say? Every one of you would say ‘I want a dark, rich, hearty roast.’ It’s what people always say when you ask them what they want in a coffee. What do you like? Dark, rich, hearty roast! What percentage of you actually like a dark, rich, hearty roast? According to Howard, somewhere between 25 and 27 percent of you. Most of you like milky, weak coffee. But you will never, ever say to someone who asks you what you want – that ‘I want a milky, weak coffee.’”
Further insights on how to interpret what people tell you:
- It cost Walmart $1.85 billion to listen to what their customers say. “This is the peril of listening to what your customers say instead of what they actually did.” – Walmart Declutters Aisles Per Customers’ Request, Then Loses $1.85 Billion In Sales and Ignore the customer experience, lose a billion dollars (Walmart case study)
- Philip Hodgson from Userfocus shows many examples where focus groups failed researchers – Is Consumer Research Losing Its Focus?
- “Now, my experience is that most of the time, people have no idea why they’re doing what they’re doing. They have no idea, so they’re going to try to make up something that makes sense.” says Clotaire Rapaille.
- In the book Yes!: 50 Scientifically Proven Ways to Be Persuasive, the authors argue that “people’s ability to understand the factors that affect their behavior is surprisingly poor.”
- Gerry McGovern writes in It’s not what people say, it’s what they do that “The worst way to design a website is to get five smart people in a room drinking lattes and posting post-it notes. […] The next worst way is to get 10 customers in a room drinking lattes and giving their opinions on the new design. That model is really, truly broken.”
- The article on Wikipedia about introspection illusion cites several studies on how bad people are at explaining their own behavior or at predicting their future attitudes.
- In his book How we decide, Jonah Lehrer shows a brilliant experiment on how your decision gets warped when you have to explain it. In the experiment, one group of students was asked to rank strawberry jams. They ranked the jams more or less the same as the Consumer Reports experts did. Another group, however, was asked to not only rank the jams but also explain their preference. This group did much worse, and actually preferred the worst jam.
- Neuropsychologist Susan Weinschenk argues that you shouldn’t believe people when they say they’d prefer certain changes in your product. They probably overestimate their future reactions.
- Joshua Porter says that “when you ask people ‘What would it take for you to use or pay for this?’ the answer you get is not reliable. Do not trust it. Don’t go implementing all the things they say because you think that’s the way to success. It’s not. It’s simply theater.”
- “There’s a lot of research out there suggesting you have no idea why you act or think the way you do. It feels awful to accept such things, so you create narratives to explain your own feelings and behavior.” – from the blog devoted to self delusion and irrational thinking, You Are Not So Smart
- Jakob Nielsen says that the “critical failing of user interviews is that you’re asking people to either remember past use or speculate on future use of a system.” – Interviewing Users
- “If I had asked people what they wanted, they would have said faster horses.” is something Henry Ford most probably never said, but he was definitely thinking along those lines. However, on the long term, not responding to customers’ needs didn’t help the company. – Henry Ford, Innovation, and That “Faster Horse” Quote