UX Diaries #2: Do users lie (sometimes) in self-reports?

UX Research

“Do users really tell you what they want in a product? Maybe not always.”

Have you ever lied about your exercise habits or sweet tooth when visiting your doctor? If yes, you are not alone. We tend to do this quite often to appear more likeable or “desirable,” particularly when the behaviour is embarrassing or socially disapproved of. This made me reflect on how this translates to product research and to what degree we can trust our users’ self-reports without further analysis.

Do users lie when asked if they like a product’s features? If yes, do they lie more when the interviewer is in the room or during self-administered questionnaires and surveys?

In UX research, we rely on two different methods when collecting data about our users: attitudinal and behavioural.

• Attitudinal research methods: Interviews, surveys & questionnaires, focus groups, workshops, diary studies, think-aloud method, etc.

• Behavioural research methods: Usability testing, A/B testing, heatmaps, click tracking, eye-tracking studies, session recordings, analytics & behavioural metrics, etc.

While this is not a research paper, and I will not try to answer these questions using statistical analysis, I will review some articles and come up with a list of conclusions by the end of this diary entry. Hopefully, these will come in handy during your next user interviews.

Why users’ answers can differ from reality.

They want to be socially desirable

 Differences between what users say and what they actually do can often be explained by social desirability bias, a tendency to answer more favourably about socially desirable behaviour to appear prosocial. For example, one might report higher church attendance or fewer embarrassing habits. Because we want to be seen as socially acceptable, this type of bias can influence users’ responses in attitudinal research.

I found that social desirability bias can hold true even in self-reported digital surveys. Interestingly, users tend to over-report socially acceptable behaviours both when interviewed and when surveyed.

They might simply not remember

 Another possible explanation for “lying” is memory error during the interview. Users rely on memory when reporting desirability or feature usage, which makes them more likely to misrepresent reality. Users don’t always intend to lie, they may simply skip or forget details while being interviewed.

Sometimes, users might genuinely believe they want a specific feature, even though data tells a different story. For instance, users might claim that they always skip tutorials, while data might show that they tend to read the initial instructions.

What if users “lie”? Should we trust them?

The answer is: sometimes. It’s counterintuitive to rely solely on interviews and surveys when users may describe a different reality. The information reported needs to be critically assessed, especially when relying on a single user expressing an opinion about a feature.

What alternative methods can we use?

• Combine behavioural/analytics data with attitudinal data and check how much they overlap.

• Use affinity mapping to identify patterns in desirability across multiple user interviews.

• To compare preferences between design options, A/B testing may be a better choice.

How can we make users less likely to lie during interviews?

• Inform users that all feedback is welcome and that you’re not judging their answers as right or wrong.

• Avoid leading questions that might influence users toward a particular answer.

• If you expect a question may trigger a socially desirable response, remind users that any answer is acceptable.

• Keep cultural differences in mind, some users may be more inclined to give “pleasing” answers.

• To reduce memory errors, ask users for concrete examples of when they used certain features.

  
  

For a deeper take on this topic, see also Kalana Dias’s Why Your Users Lie (And How UX Research Reveals the Truth).