Usability testing is a widely used research method to collect detailed and direct qualitative user feedback about your product (or website, mobile application, anything…) What is it exactly and how does it work? Why is it important for Data Analysts and even for Data Scientists? How can you do it by yourself? By the end of this article all these questions will be answered!

What is usability testing?

Note: In this article I’ll use an example of a website/online product, but you can replace it with “mobile application”, “e-commerce store”, “new feature” or anything.

So what is usability testing? It’s as simple as inviting a user (or a potential user) from your target audience – then showing him/her your website, giving specific tasks and for around 30 mins monitoring what she/he is doing.

If you do it right, you will get a bunch of useful and actionable insights! You will be surprised how often the feature you consider to be one of the most straight-forward things on your site turns to be the most confusing for your target audience. And that’s the point here: to see and feel the actual pain of your users!

It’s worth mentioning, that usability testing is a qualitative method. This means, that you will do it on a smaller sample size (~5 tests per round). Thus you can’t handle the information you collect here as “statistically significant” data, but still, it will give you priceless insights and good hunches for further analyses.

simple usability testing on mobile

simple usability testing on mobile (source: Flickr of City University Interaction Lab)

Why usability testing is important for data professionals?

You might ask: “I’m a data analyst. I work with quantitative methods and not with qualitatives.” As I’ve already described on my previous article (The Online Research Framework I use at Data36), as a data professional you should not narrow down your research arsenal for data mining only. Qualitative methods like usability testing will give you information, that you would never get with any type of analyses.

Example1 (for data analysts):
If you run a website heatmap you might see that people do not click at your conversion button (let’ say only 0.1% of them). That’s a fact supported by data. But you still don’t know, why didn’t they click! Don’t they see it? Don’t they get it? Don’t they care about it?

Example2 (for data scientists):
You have discovered a direct correlation between churn and a specific product feature. Great! You see that if new users use this feature, they will churn in 3 days. If they don’t use it, they will churn only in 30 days. But why is it so? Are they unsatisfied with the feature? Are they looking for something else? Does this feature communicate something wrong about your product? We won’t know until we don’t ask the users.

Usually, I like to say that quantitative methods are great in discovering the answers for “What”, “When”, “Who”, “How”, but only qualitative methods will give you info about the “Why”.

Note: if you are a UX-person, don’t get too excited, it’s also true the other way around. Using qualitative methods only is just as narrow-minded, as using analytics only.


Qualitative researches and Data Analyses go hand in hand

If you want to learn, how you can fit usability testing into your analytics efforts and A/B testing processes, I highly recommend this article: Online Research Framework.

How to run your first usability test?

Are you convinced? Good! Then it’s time to run your first usability test! Don’t worry, it is fairly easy. First things first: invite some users. Where should you meet? In your office preferably – but in some cases a coffee house can do the job. Or even Skype or Google Hangouts conversations (eg. if you are running an international business). Then sit with each of the users – one at a time! – and go through your website.

Before you start it’s good to ask the participants to say out loud everything what’s on their mind during the test. Of course, you can ask questions too. (What is a good or bad question? I’ll get back to that below.)

It will help you a lot if you don’t just ask them to “go through your website”, but prepare with a scenario as well. Something like “imagine that you are going to Holidays – you have a $1000 budget and you would like to book your flights and hotels”. If you go this way, you can avoid the awkward situation of the testee clicking around on un-important subpages or get stuck after 3 minutes – with the question: “okay, now what should I do?”.

I also recommend to record the sessions and take notes.

Aaand.. that would be your first Usability Test. These are the principles at least. Of course there are a lot of best practices here. I’ll give you some of them to make things smoother for you.

Usability testing best (and worst) practices

Most of these best practices are to not let yourself be biased. Also to avoid inconveniences and make follow-ups easier. Let’s see:

1) One user at a time.

It should be self-explanatory, but if it is not: always test with one person at a time. This way you can make sure, that participants won’t influence each other.

2) Record your sessions.

I’ve already mentioned this, but I didn’t tell why. It’s good for 2 things:

  • A) You can watch again your test later, so you won’t forget the key-findings.
  • B) You can show the test to anyone at the company. Presenting a chart to a designer and saying that “your design sucks” (okay, maybe it’s better with different wording) could be convincing. But it’s much more powerful, if you can support your numbers by a recorded real life use case, where an actual human being is getting stuck with UX issues.

The easiest way to record a session is to use a screen-recorder application. There are plenty available solutions on Mac, iPhone, Windows and Android too. (Eg. on Mac and iPhone, I use QuickTime Player. And sometimes Camtasia.) Some people like to record the face expressions of the user. I found it more disturbing, than useful so I usually don’t recommend it.

Note: some UX professionals do face recordings, because the face sometimes tell a different (more honest) story, than the words!

usability testing record your sessions!

Record your sessions!

3) Always try to come up with a scenario.

It will just make things easier and more natural. The scenario should be a common or expected use case of your product.

4) Before the test – explain what usability testing is.

Most probably your applicants have never been a part of a usability test. Thus they don’t know what it is. This can be uncomfortable, but you want to have relaxed people in front of you (as more natural behaviour means more accurate test results), so explain them why are you doing the session and what will it look like.

5) It’s not a user test. It’s a usability test.

Also explain before the session, that you are not testing them – you are testing the product. If they got stuck with a feature, that’s never their fault. It’s the product’s fault.
Back in the days, I’ve forgotten to mention this before a few of my tests. Boy, I don’t wish anyone those awkward situations, when the testee continuously says “sorry”, when he/she doesn’t find a button or doesn’t understand the feature of a crappy prototype.

6) Encourage them to say out loud what they think.

Again: you will get the most out of a usability test, if you learn, what’s in your users’ head. However this is not obvious for them. So encourage them to say out loud their thoughts!

7) Try to stay passive.

Once you have given the scenario, try not to influence your participant. Try to stay passive and only talk, when she/he asks something.

8) Use open-ended questions!

Yes, you are allowed to ask questions! Your favorite question should be “why”. “Why did you click here?” “Why did you do this?” “Why did your prefer this and not that?”
Try to avoid closed-ended questions and biased questions. Worst questions are:
“Do you like this website?” (Who would say to the creator of a website, that it’s shitty?)
“Should this button be red or green?”
“Would you share this on Facebook?”

9) Be nice!

Well, yeah, just in general: be nice to people and they will be nice to you. (Okay, life-coaching session is over.)
But be super nice, when they are coming to you to do a usability test. Remember: this test situation will be out of their comfort zone. Again: the more stressful they are, the more biased your test will be. So try to help them to relax. Start with a bit of small talking, bring a cup of tea, etc… But careful: don’t be too nice either, because in that case the user would have a hard time to give you a negative feedback about your product.

10) Never do usability testing with developers or designers.

I did that a few times. It was a horrible experience. Designers and developers are looking at your website from a whole different perspective, than your normal users. They will come up with random ideas how your buttons should look. They will look at the code, at the responsivity and other stuff, that normal users would never do. These tests can go straight to the garbage.

11) Do ~5 tests per round!

There are different schools on the number of users per test round. But according to Nielsen Norman Group’s research the best is to invite no more than 5 users. They say:

“Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford.”

usability testing do 5 usability tests


12) Make notes!

I usually make notes right after the sessions. So I will remember the top-findings. Once I’m done with all the 5 sessions, I can prioritize these and decide the next step: quick fix (for bugs), further analyses (for possible UX issues) or ignore (for smaller things that occurred only once).

How to hire users for usability tests? Where to find them?

This is one of the top questions I usually get, when I talk about Usability Testing on conferences: how to find users for your tests?

There are 3 methods I use:

  1. If you want to test something (eg. new feature) on your existing user base , you can just simply reach out to them. Send an email to 100 of them, offer free pro features for a few months plus branded T-shirts/sunglasses maybe.
  2. If you want to test on potential users, go where your potential users are. Eg. a few years ago I’ve done usability tests for a body-building e-commerce store. I did 2 things: a) went down to the gym and put some posters on the wall (with the permission of the gym-owner of course) b) posted on body-building Facebook groups. We offered to the applicants 20$ coupons on their next purchase.
  3. If you want to test on potential users and you have no idea how to find them – try or They can hire you users from almost any user segment for 49$/test.

A pro tip for the first 2 cases: every time I hire users, I hire them via a survey-like subscription form. Here I can ask some specific questions, that helps me filtering out applicants, who I don’t want to test with (eg. developers or designers as I’ve mentioned before).

When to do it?

If you have an online product – just in general – to do usability testing is a great idea at any time. You know, it’s always good to get direct feedback from your users. But it’s especially recommended to do it:

  • If you start a CRO (conversion-rate-optimization) project
  • If you are about to release a new feature
  • If you are about to redesign your website
  • If you are about to release a brand new product
  • If you have never done it before

Here’s an article on how to fit this method in your whole research project: The Online Research Framework.

online research framework

the Online Research Framework – 6 steps

Plus it’s good to know, that greater online businesses (~50+ people companies) usually have a dedicated UX research team, that is doing Usability tests on a regular basis. If you have a bigger project, I can also recommend to hire an experienced UX researcher to do the job as a pro!

Conclusion, Disclaimers, Limitations

As you can see Usability testing is an easy and extremely useful method. I can tell you, – as I’m a data analyst – when I was first hearing about this concept, I was a bit sceptical… I thought that these kind of qualitative researches are always biased by default and on the top of that, they will never give actionable information because of the small sample size. But I was wrong. Recently I’m using usability testing almost every time, when I start a research project.

Assuming that you are more into quantitative things too, I’d say this practical high-level summary will be good enough for you! Now, go and let’s arrange your first usability test! 😉
If you want to learn more, check professional UX blogs, there are plenty.

And if you want to be notified first about new content on Data36 (like articles, videos, handbooks, webinars, etc.), sign up for my Newsletter!

ps. and for a closing fun here’s how to do usability testing of fruits – it’s half funny, but you can still learn a lot from it:

Tomi Mester