Usability testing is a popular research method for collecting detailed and direct user feedback about your online product, website or mobile application.
What is it exactly and how does it work? Why is it important for data analysts and data scientists? How can you run a proper website usability analysis by yourself? What are the best usability testing questions?
By the end of this article all these questions will be answered!
What is usability testing?
Note: In this article I’ll use the example of a website. But everything here applies to mobile applications, e-commerce businesses and online products, too.
So what is usability testing? It’s as simple as inviting a user (or potential user) from your (target) audience – then showing her your website, giving her specific instructions and, for around 30 minutes, monitoring what she’s doing.
If you do it right, you will get a bunch of useful and actionable insights. You will be surprised how often a button or a description that you consider to be one of the most straightforward things on your website turns out to be the most confusing one for your audience. And that’s the point here: to see and feel the actual pain of your users!
It’s worth mentioning that usability testing is a qualitative research method. You will always work with a smaller sample size: around 5 usability tests per research round.
Obviously, this means that it’s not like data analysis. You can’t consider the information you collect here as “statistically significant.” Still, it will give you priceless insights and good hunches for further analyses.
Why are usability testing methods important for data scientists, too?
You might think: “I’m a data scientist. I work with data and not by chit-chatting with users.” As I’ve already described in my previous article (The Online Research Framework I use at Data36), as a data professional you shouldn’t narrow down your research arsenal to data research only. Qualitative methods like usability testing will give you in-depth information that you would never get from raw data.
Let me give you two examples.
Example #1 (for data analysts):
You run a website heatmap and you see that people do not click your “To the Cart” button (let’s say only 0.1% of them). That’s a fact supported by data. But you still don’t know why they didn’t click! Don’t they see it? Don’t they get it? Don’t they care about it?
Example #2 (for data scientists):
You have discovered a direct correlation between user retention and a specific product feature. Great! You see that if new users use this given feature, they will churn in an average of 3 days. If they don’t use it, the average goes up to 30 days. But why is that? Are they dissatisfied with the feature? Are they looking for something else? Does this feature communicate something wrong with your service? We won’t know until we ask the users.
Are you getting it?
Data can help you answer many questions:
- What do your users do?
- When are they doing it?
- How are they doing it?
- …
But data will never be able to answer the “why” questions. And if you think about it, quite often, that would be the real game-changer. Those “why” questions could help you to understand your audience’s reasons, motivations and desires.
And that’s why usability testing methods are so useful. An in-depth discussion with a few users will reveal many, many unknown things.
Note: if you are a UX person who reads this, don’t get too excited! 🙂 This goes the other way around, as well. Using usability tests only is just as narrow-minded as using data only.
How to run your first usability test?
Are you convinced? Good! Then it’s time to run your first usability tests! It’s not too difficult.
First things first: invite a few users!
Where should you meet? In your office, preferably, but a coffee house can also do the job. Doing it online via Skype or Google Hangouts is quite common, too. (E.g. if you are running an international business but don’t want to travel.)
Sit down with each user – one at a time – and go through your website.
Before you start it’s good to ask the participants to say out loud everything that’s on their mind during the test. Of course, you can ask questions, too. (What are the good and bad usability testing questions? I’ll get back to that soon.)
It will help a lot if you don’t ask the users to simply “go through the website”. That’s too abstract. Give them a real-life-like scenario instead! Something like “imagine that you are going on vacation – you have a $1000 budget and you would like to book your flights and hotels!” By using a scenario, you can avoid the awkward situation of the tester clicking around on unimportant subpages or getting stuck after 3 minutes – with the question: “okay, so what should I do?”.
I also recommend recording the sessions and taking notes.
Aaand.. that’s it, your first website usability test. These are the key principles, at least. Of course, there are a lot of best practices to make this more efficient. Let’s go through a few of these.
Usability testing best practices
One of the main problems with usability tests is that they can be biased pretty easily. The best practices I provide here are about preventing that.
Let’s see!
1) One user at a time.
It should be self-explanatory, but if it is not: always test with one person at a time. This way you can make sure that participants won’t influence each other.
2) Record your sessions.
I’ve already mentioned this, but I didn’t say why is it good for you.
2 things:
- You can watch your usability tests again later, so it will be easier to recall your key findings.
- You can present the results to anyone at the company. Presenting a chart to a designer could be convincing… But it’s much more powerful if you can support your numbers with a video-recorded real life user story, where an actual human being is getting stuck because of UX issues.
The easiest way to record a session is to use a screen-recorder application. There are plenty of available solutions on Mac, iPhone, Windows and Android. (I use Camtasia Studio.)
Note: Some UX professionals like to record the facial expressions of the users, too. The face sometimes tells a different (more honest) story than words. I personally don’t do face recordings because I found it more disturbing than useful — but that’s my opinion only…
3) Always try to come up with a scenario.
It will just make the whole interview situation more natural and life-like. The scenario should be a common or expected use case of your product.
4) Before the test, explain what usability testing is.
Most likely your testers have never been a part of a usability test before. So they don’t know what it is and how it works. That can make them feel very uncomfortable. And you want to work with relaxed people (as more natural behaviour means less biased test results), so explain to them what they can expect from a usability test, how it works, etc.
5) It’s not a *user* test. It’s a *usability* test.
Before the session, also explain that you are not testing the testers – you are testing the product. If they get stuck with a feature during the interview, that’s never their fault. It’s the product’s fault.
Back in the day, I forgot to mention this before a few of my usability tests. Boy, I don’t wish on anyone those awkward situations when the test user says “sorry” every time she can’t find a button or doesn’t understand a feature of a crappy prototype.
6) Encourage them to say out loud what they think.
Again: you will get the most out of a website usability test if you understand what’s in your users’ head. This is not obvious to them though. So encourage them to say their thoughts out loud!
7) Try to stay quiet.
Once you have given the scenario, try not to influence your participant. Try to stay passive and only speak when the user asks something.
8) Use open-ended questions!
Yes, you are allowed to ask questions! Your favorite question should be “why.” “Why did you click here?” “Why did you do this?” “Why did your prefer this and not that?”
Try to avoid closed-ended questions and biased questions! The worst questions are:
“Do you like this website?” (Who would say to the creator of a website that it’s crap, right?)
“Should this button be red or green?” (It’s not her job to know that.)
“Would you share this on Facebook?” (She might lie – even unintentionally.)
9) Be nice!
Well, yeah, just in general: be nice to people and they will be nice to you.
But be super nice when they are coming to you to participate in a usability test. Remember, this situation will be out of their comfort zone. Again: the more stressed they are, the more biased your test results will be. So try to help them to relax. Start with a bit of small talk, bring a cup of tea, etc… But careful: don’t be too nice either, because in that case the user would have a hard time to give you negative feedback (which you are looking for, too!)
10) Never do usability tests with developers or designers.
I did that a few times. It was horrible.
Designers and developers are looking at your website from a whole different angle than “normal” users. They will come up with random ideas for how your buttons should look. They will look at your code, at the responsivity of your website, and a bunch of other stuff that normal users would never do. These test sessions are just a complete waste of time.
11) Do ~5 tests per round!
There are different opinions about the number of tests per test round. But according to research from the Nielsen Norman Group, the best tactic is to invite no more than 5 users in one iteration. They say:
“Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users…”
12) Take notes!
I usually take notes right after each session so I will remember the top findings. Once I’m done with all 5 sessions, I can prioritize these and decide the next step: quick fix (for bugs), further website usability analysis (for possible UX issues) or ignore (for smaller things that occurred only once).
How to hire users for usability tests? Where to find them?
This is one of the top questions I usually get in workshops: how can you find users for your tests?
There are 3 methods I use:
- If you want to test something (e.g. a new product feature) with your existing user base, you can simply reach out to them. Send an email to 100 of them and offer free pro features for a few months, plus branded gifts/T-shirts/sunglasses, to participate in a 30-minute test.
- If you want to test on potential users, go where your potential users are. E.g. a few years ago I did website usability analysis for a body-building e-commerce shop. I did 2 things: a) went down to the gym and left a few flyers there (with the permission of the gym owner, of course) b) posted to a body-building Facebook group. We offered $20 coupons for the applicants off their next purchase in the e-shop.
- If you want to test with potential users and you have absolutely no idea where to find them, try usertesting.com or userfeel.com. They can hire you users from almost any user segment for $49 / test.
A pro tip for the first 2 cases: every time I hire users, I hire them via a survey-like form. Here I ask specific questions that helps me filter out applicants who I don’t want to work with in the tests (eg. developers or designers as explained above).
When to do it?
If you have an online product – just in general – running a few usability tests is a great idea at any time. It’s always great to get direct feedback from your users. But it’s especially recommended to do it:
- when you start a CRO (conversion-rate-optimization) project
- when you are about to release a new feature
- when you are about to redesign your website
- when you are about to release a brand new product
- if you have never done it before
Here’s an article on how to fit this method into your whole research project: The Online Research Framework.
It’s good to know that bigger online businesses (~50+ people companies) usually have a dedicated UX research team, too, that’s responsible for running usability tests on a regular basis. If you have a bigger project but you can’t afford a UX team, I recommend hiring a professional freelancer UX researcher to do the job!
Conclusion, Disclaimers, Limitations
As you can see, usability testing is an easy and extremely useful method. I can tell you – as a data analyst – when I first heard about it, I was a bit sceptical… I thought that these kind of qualitative research methods are always biased, and on top of that, they will never give actionable information because of the small sample size.
But I was wrong.
Nowadays, I run usability tests almost every time I start a research project.
Assuming that you are a data scientist, I’d say this practical high-level summary will be a good start for you! Now, go and arrange your first usability test!
- If you want to learn more about how to become a data scientist, take my 50-minute video course: How to Become a Data Scientist. (It’s free!)
- Also check out my 6-week online course: The Junior Data Scientist’s First Month video course.
Cheers,
Tomi Mester
Cheers,
Tomi Mester
ps. and just for fun, here’s how to do usability testing of fruits – it’s half joking, but you can still learn a lot from it:
