The term “user testing” is one I’ve seen make the rounds quite a bit in the product management world. Mind you, it’s actually got a search volume of over 8,000 hits per month.

But did you know that “user testing” isn’t actually the correct term?

When thinking of user testing, many people actually mean usability testing. If you’re lost about the difference, don’t worry!

In this blog post we’ll explore the correct terminology, how to go about executing product testing, and how you can get ready for your next user session.

Let’s do this!

TLDR

• User research is used when trying to find product-market fit and do further research on behaviours and frustrations.
• Usability testing is used to understand if the interactions designed on a particular feature or area of your product helps the user build better habits and complete tasks as expected.
• Benefits of user research vs usability testing: how they help find product-market fit and design better products.
• Preparing for interviews: Define who, what, why, when, and how early on — and set a measurement for success.

Understanding Terminology

Testing is where you evaluate your product or service with real users and create human-centered products.

This process enables you to understand your target audience’s behavior when interacting with your product, providing valuable insights for the development process.

User research vs usability testing

The terms sound very similar which is why they tend to get mixed up, so let’s analyze this further.

User research is done when you’re trying to understand your users better. This may include things like common behaviors, patterns, frustrations, and goals.

Usability testing is done when you’re trying to see how a user behaves with certain interactions. This is generally done when designing a new feature and you want to know if the design makes sense for the user, while further assessing if it’s helping them reach their desired outcomes.

Both will give you insights into the general behaviors of your users, but one focuses on understanding them while the other focuses on understanding their interactions with your product.

User testing combines both, but it’s not the correct term. You’re not testing your users — you are either learning about them or testing if your product helps them achieve a particular task.

Cool — now that we’ve gotten that out of the way, let’s talk about the benefits of each.

Both types of interviews can give you a wealth of insight as to the people that you’re solving problems for. It’s important to understand people’s motivations, outcomes, and frustrations in order to design products that will serve them well.

Benefits of user research

1. Find product-market fit

Finding product-market fit is an integral part of product success. If your product doesn’t solve a real-world problem, will it even go anywhere? (I’m looking at you, Juicero.)

It’s important to understand who you are building for so that you can target those people better.

2. Understand your different user groups

PMF is a continuos process. If you found it, that’s awesome!

But don’t settle for what you think you know. It’s likely that within your existing audience, there are several sub-groups you need to consider. Think of things like education maturity, company growth status, and even roles within an organization.

All these people will act and think differently based on their own outcomes, and getting to know them better will help you define how to best build an app that caters to their needs.

3. Identify possible barriers to adoption

User research can help you understand how your audience perceives your particular industry, product, or brand. It will even give you insights into purchasing adoption habits, processes, and how to potentially overcome them.

Think of large enterprises with lengthy processes — the more you are prepared to provide them with smooth onboarding and remove some of those barriers from the beginning, the more likely they are to choose your product.

4. Define strategic timing

There is a reason product managers are constantly advocating to speak to users.

It’s not just about getting general feedback, or even feedback on specific features — it’s to gain wider context as to when it is potentially beneficial to pivot.

You might be making assumptions that you’re on the right track, when in fact it might be time to change gears and focus on a different problem.

Talking to people will give you the insight that you’re looking for.

Benefits of usability testing

Once you have an understanding of who you are building things for, what their outcomes are, and have come up with some solutions, it’s time to test your designs.

Usability testing is there to help you figure out if you’re on the right path, and where to potentially make improvements to the solutions you’re coming up with.

1. Provide unbiased feedback

We all have a level of emotional attachment to the things we build. That unfortunately also leads to a level of bias that what we’ve designed is right.

To truly build an unbiased, accessible, and helpful product with the largest impact, it’s important to talk to people who have a different level of attachment to yours.

2. Understand mental models

If there’s anything I’ve learned in my years in product, it’s that different audiences have different ways of interacting with technology (and this is why user research is so important!)

Once you understand your users, you can then also understand how they expect to interact with your app, and help them complete tasks with ease rather than with added frustration.

3. Find points of friction and increase adoption

Just like user research, usability testing of your design can help you identify potential points of friction.

This will help you weed out any problems from the very beginning, and make sure that when exposed to a larger number of users, the solution is clear, precise, and helps users reach their desired outcome.

Preparing user interviews

Before you start lining up people for calls, make sure that you fully understand what it is that you’re doing and why.

You don’t want to walk in unprepared.

Regardless of whether I’m running a user research interview or a usability test, I like to make sure I have all the information upfront for the entire product team.

Not only does this create alignment and transparency, but it sets the outcomes for what it is we are wanting to learn.

I generally start with a short statement that helps everyone understand who, what, why (and later on how and where) to begin with internal alignment:

Let’s take a look at these sections a bit more in-depth:

Who

Define who it is you’re interviewing. You may think that you have a set audience, but within all target audiences, there will inevitably be different sub-groups.

Are the people that you are talking to…

• Senior leaders?
• Decision-makers?
• Those with purchasing power?
• Are they trial users, paid users, power users, or casual users?

Depending on what it is you are looking to find out, the audience needs to align with the outcome.

What

Product management 101: define what problem you’re trying to solve!

If you’re not quite sure where to get started, this product problem outline is a great kick-off point.

When it comes to interviews, define at this point what it is you are wanting to learn when jumping on a call with a customer, and how you would identify success. (When doing usability testing, defining success criteria is a must!)

Why

This is heavily tied to the ‘what’ of the research.

• Why is this information useful to you?
• What will you do with it once you have it?
• How will it impact your next decision?

Executing user interviews

Now that you’ve got the who, what, and why — it’s time to define how and where.

Taking our previous statement on who, what, and why, we can then expand on it, adding how and where:

How

Getting the ‘how’ part is just as important as everything else, as it refers to the actual execution of the interview.

A few things you’ll need to define at this stage are:

• The script
  Identify and outline key questions upfront. This will keep you from deviating and ensure you’re sticking to the outcome you want to reach. Having the same set of questions will also set a standard that will keep the study unbiased.
  That said, remember to still be open to the general flow of the conversation. It’s likely you will need to ask ‘why’ or ‘can you tell me more about that’ a few times to find the root cause of a particular problem — and know that this is also ok! Sometimes interesting points come up when you least expect them.
• Don’t ask leading questions
  Falling into the trap of leading questions is easy, it can happen to all of us. Make sure you aren’t accidentally prompting the user to answer in a particular way, rather keep all questions impartial and neutral.
• Be respectful of other people’s time
  It’s important to be conscious of other people’s time. I personally love running interviews, and I know how easy it is to get excited about learning from others. But one must also be respectful of the workload others may have on their plate.
  In general, try to time it so that the interview is no longer than 30 minutes. If you’re not quite sure, do a test interview with an internal user (like someone from your sales team) to help you figure out which questions could possibly be removed. At the very least you’ll learn something!
• Delegate roles
  I never do interviews on my own. Firstly, this is because I want to make sure I’m not accidentally introducing my own bias. Yes, there is a script at hand, and that should prevent this for the most part, but having a moderator is an extra guard rail to ensure that the interview goes smoothly.
  
  Second, it’s important to have a note-taker. Running the interview and taking down notes at the same time will be distracting for both yourself and the participant. While you may be recording the session (and you should), a note-taker can write down certain reactions and observations that may be overlooked otherwise when watching back the video.

Where

“Where” refers to both where you will find and contact participants, as well as the method in which you will conduct the interview.

Depending on the type of interview you’re running, you may want to ask your direct customers, those that are leads, or even people outside of your organization but still within the same industry that you’re catering to.

There’s no wrong or right here — it’s really about what you’re looking to learn.

A few tips:

• Userpilot: For collecting survey data in-app from existing users and tracking product usage
• Typeform: For quick surveys and to sign up participants.
• Calendly or Hubspot Calendar: To allow people to select a time/date of their choice for the session.
• Linkedin, Slack, or Twitter: When looking for industry-related participants.
• TLDV.io: To record user sessions with transcripts and annotations.
• Notion: To write down notes, analysis, and research guides.

Bonus: ‘How many’ ?

The question of ‘how many’ people you should interview is a tricky one.

On the one hand, you certainly want to have enough data, on the other hand, the interview process can be long and extremely time-consuming to do.

Good news, there is actually a way around it.

Jason Nielson, from the Nielsen Group explains the magic rule of 5.

Summarizing your findings

Once you’re done, it’s time to summarize your findings.

I hate to say this, but the truth is no one else is going to read through pages of research and watch videos just to try and keep up. The best thing you can do for your team is provide them with the key points and learnings and how these link back to your desired outcomes.

• What did you learn?
• How did the findings challenge your hypothesis?
• Are there any actionable takeaways?
• What can you do with this information moving forward?

Of course, it goes without saying that you should discuss your findings as a team. Talk about what you have learned and how it can help you make informed decisions moving forward.

Happy interviewing! ????