User testing is a useful qualitative research tool and a generally great way of providing valuable insights into how real users interact with a website or product. It helps designers or anyone who works in UX or CRO identify any issues or pain points that customers may encounter, leading to a better overall user experience.
In this article, we will go over how to plan, launch, and analyze web user testing, including the tools that are good for user testing, why it can be expensive, and how much data/testers you need.
One of the first things to consider when planning web user testing is the platform that you will use. There are a variety of tools available, each with its own strengths and weaknesses. Most of these tools work by the same logic: they create a pool of testers you can recruit from, or you can set up the test and recruit your testers from somewhere else. This is usually a much cheaper option but requires more effort.
Which option to go with is highly dependent on whether or not you have a large enough email list or enough traffic on your website to be able to recruit your own testers in exchange for some sort of incentive. Even if you do, there are some downsides to this approach:
Because of these challenges, most companies tend to pay for full-service platforms and make use of their pool of testers. The logistics and rewards for tests have been fleshed out and you’ll be able to get your results back very quickly most of the time if your targeting isn’t too nitpicky. Another pro for the full-service platforms is that most of them are willing to compensate for bad tests. This means that if a tester's audio quality is really bad or they skip most of your tasks, most respectable platforms will give you a free test in return to run it again.
Within the tool, you usually have the option to launch a moderated or an unmoderated test. For the purposes of this article, we’ll be focusing on unmoderated testing since this is already a lot of work, and having a moderator present for each session adds to that significantly and creates the need to schedule the sessions.
A good testing platform will also offer the capability of doing tests on different devices like desktop vs mobile. This is important because the user experience between mobile and desktop devices is fundamentally different, so you’ll want to research those separately to draw any reasonable conclusions.
Some tools to check out for user testing:
One question that often comes up when considering user testing is why the cost can be so high. There are a few reasons for this:
The pricing models of the tools vary so much that it’s really difficult to estimate a ballpark figure of how much a user testing project will cost you. Some tools allow you to “pay as you go” starting from around 30 dollars per test, but then there are usually time limitations on how long the test can be, etc. Some platforms require demos before they tell you anything about pricing, some require a monthly or yearly subscription fee on top of charging you separately for each test, and sometimes you need to purchase large buckets of credits at a time.
I would even go so far as to say that one of the more annoying and complicated parts of launching a good user testing program is to figure out which tool to use just because the pricing models have gotten so complicated and a lot of them are designed to lock you in for a longer period of time. A little bit of research and comparison to figure out the best tool for you is still worth it for the insights you will get from your user testing project.
The answer to this question will depend on your specific goals and the complexity of your website or product. In general, it is recommended to test with at least 5-10 testers to get a good understanding of any issues or pain points. Read through this great article by Jacob Nielsen on “Why you only need to test with 5 users” before you run and launch a very expensive 100-user test that will take ages to analyze just because someone told you that you need statistical significance.
The other aspect to consider with user testing, as just mentioned, is the analysis. An average good quality test that takes the user through the entire web funnel will take around 20-40 minutes. If you launch 5 mobile tests and 5 desktop tests and they average a half hour each, then this will add up to 5 hours of watch time at least.
To be fair, most user testing platforms offer the option to watch the videos at different speeds so you can play them all on 1.5 or 2x speed to make life easier, but the reality of this kind of analysis is that you need to pause, take notes, take a break, really deeply think about what you saw and what it means, and then usually compile a report that somehow brings this project together and conveys the findings to the people that didn’t watch these videos but still need the insights from it.
Before you start putting together the task list, spend a bit of time figuring out the overall purpose of your user testing project. Are you trying to understand the entire shopping flow, or do you need to focus on a specific area on the site that has shown to be problematic? Are you trying to inform an experimentation program by gathering ideas for new test hypotheses? Perhaps it's something more unique to your business case entirely?
When planning the actual tasks for user testing, it is important to keep in mind the specific goals of the test and the overall user journey. Some specific tasks to consider might include navigating to a specific page, completing a form, or purchasing a product. It is also important to include both specific and general tasks to get a well-rounded understanding of the user experience.
Some sources will tell you to create arbitrary success metrics to decide whether or not a tester “passed or failed” the test. This to me seems extremely counterintuitive and defeats the whole purpose of a qualitative research project. Your purpose should be to find out what you don't know. There should be no wrong answers for the testers. As mentioned you should absolutely include more specific tasks for the testers if there is an area of the site you're specifically interested in, but the overall sentiment should be as open-minded as possible. It’s way more interesting to see how and why a user failed a task than to count the number of users who failed.
Most platforms give you the option to either require testers to have cameras on or off. Unless you’re super into reading facial expressions, there’s not much value in having the camera on for a moderated test. Make sure you emphasize in the intro though that you want them to narrate absolutely everything they’re doing, to “think out loud” and that there are no wrong answers.
Here’s a list of potential tasks you might want to test on a typical e-commerce site:
The shorter your funnel, the harder it will be to do proper user testing. If you work with a SaaS website instead or any type of business where the signup flow is incredibly simple, you might benefit more from exit polls/on-site surveys that are triggered to launch right before the customer is about to abandon the site.
This is kind of a tough one to balance. On the one hand, you want someone who could at least be a potential customer for your brand, but having a tester who is already familiar with your brand and site is not really helpful either. Often we create the first task as a short recognition task. We ask the testers to look around for a couple of seconds and describe what the company does, sells, and stands for. This is also called an impression test and the purpose is to see if new customers are able to grasp what you’re offering or if your value prop, landing, or home page needs a lot more clarity.
Most testing platforms will let you choose from a variety of targeting options such as location, age, level of income, education, and gender. This can help ensure that you are testing with a diverse group of users who represent your target audience. It truly does depend on your demographic.
If your products are exclusively targeted at women and you know that you have no male customers, then there's no point to test with male testers. If you sell luxury products then it would be wise to select an appropriate income level within the targeting. If your core demographic is seniors, then getting a bunch of teenagers to user test your site isn’t going to be very helpful.
Most platforms also offer the option to add something called the qualifying question. This means that if you can't find the right options within their targeting rules you can set up a disqualifying question yourself. Let's say your company sells dog food for example and there is no targeting rule for pet owners, you can launch the test with the first question “do you have a dog?”.
Outside of cases like these, we recommend not trying to target your exact ideal customer profile. User testing isn’t meant to gather perceptions or opinions, it’s more about observing how people interact with your site. There are better ways to deepen your understanding of your customer’s mindset, like customer surveys, interviews, or even on-site polls, which are done with your actual customers and users rather than people recruited from a testing platform.
There are two mental layers you need to apply while doing user test analysis. The first layer is noting everything that the tester says, the other is noting everything they actually do. Spoiler alert - the two don’t actually always add up. While user feedback can be valuable, it’s important to also observe actual behavior to get a complete understanding of the user experience.
Most humans are incredibly biased, and that includes your user testers. Testers may want to be liked or may feel pressure to perform well because they are getting paid for their time. We see this all of the time. They can barely get through or understand the task, and they’re struggling a lot, but the thoughts they choose to articulate in the moment are the opposite and paint your site in a non-problematic positive way.
To mitigate this, you might find it helpful (or just interesting?) to go through the FAQ for testers and understand the payment terms and expectations. You might also end up with a couple of “serial testers” as we call them. These are the people who try to do this type of testing for a living and spend their days on these testing platforms. They’ll try to get through the test as fast as possible, leaving you with little to no actual insight about your website experience. In this case, you should absolutely reach out to the platform support again and get your test replaced.
Some platforms have the feature of adding notes and annotations within the app, but you can also just take notes on whatever notes app you are comfortable with. I personally choose to use a Google Sheet and the unclean notes look something like this:
A separate sheet for each device, testers in the first column, and each task in the first row.
When taking notes during the user testing analysis, it is important to write down everything you see without censoring yourself. This will help you capture any issues or pain points that users may encounter. This type of template will give you the opportunity to look at issues for each tester per task and later compare the common patterns within each task. For example, if 4 out of 5 testers didn't get through checkout, you can go to your notes in the checkout task column, and it will be immediately clear.
When analyzing the results, it is helpful to look for common patterns and issues as well as singular issues. Even if a specific issue only arises with one tester, it is still important to take it seriously as it may be indicative of a larger issue. It is also important to give testers enough time to complete the tasks, typically 30-45 minutes. This will allow them to take their time and not feel rushed through the process. To the analysis sheet, you can add the duration of each test for more clarity. I also like to note which was the longest task for each tester individually.
Your unfiltered notes are usually not enough for a presentation or a report though that can clearly communicate the overall issues that came out of this research method. As a last step of the analysis, you can put together a report that highlights the major issues and breaks down everything by task. A couple of examples from the full Koalatative User Testing Report:
It’s good to have a breakdown of the gathered results and remind everyone why you didn't launch a hundred tests as we mentioned in the beginning.
Every once in a while testers will spit out pure gold as direct quotes and answers to the questions that made you start this project in the first place. Most user testing platforms now offer transcription of the video as well so it’s fairly easy to extract those quotes and add the best ones to your report to really drive the point home. Here's an example of how we communicate the best quotes in our report:
Each task or activity that your testers go through should have its own section in your report where you break down the issues by device. Keep in mind that we’re not trying to find significance here by saying 3 out of 5 doing something is worse than 2 out of 5 and so on. This is just a useful simple way to break down the issues and communicate them to the people in your team who didn’t watch 5 hours of tester videos. Here's an example:
For added value, you can also record short clips of testers struggling with something specific that you want to mention and add those to your report as gifs.
User testing can be a valuable tool for uncovering blunders in the user experience of a website or product. By using the right tools, crafting the right set of tasks, and carefully analyzing the results, you can identify and address any issues or pain points that users may encounter.
It’s definitely not the easiest form of qualitative research - this is also one of the reasons why a lot of companies don’t invest in this method - but by taking the time to properly plan, launch, analyze, and present the results you gathered through user testing, you can ensure that your website or product is the best it can be for your customers.
We'll send you an email when we publish new content