Usability Study

Key Points

• Identify problems with your product.
• Discover opportunities to improve the product.

What are they and Why use them

A usability study evaluates a participant’s ability to use a product or service by observing them. The participants should be representative of those who would normally use the product to determine how effective and efficient it is to use, how easy it was to learn to use the product, and to identify key areas of concerns that participants have with the product. Doing so provides strong evidence of how close the product is to being ready for deployment, and can also identify whether the product is providing the right service to the user to establish its utility. The output of a usability study includes both qualitative (the users’ opinions of the product including likes, dislikes, and desired features) and quantitative results (successful tasks, completion times, error rates, etc.).

When to use them

Conducting a usability study early on during the development stage allows for issues to be identified and solutions to resolve them to be applied while the cost of implementation is low. In the image of the human factors and engineering process, the usability study is shown as being conducted as soon as a working prototype is developed, and in some cases the prototype can be very low fidelity. For example, if you were developing a health application on a mobile device, a wireframe model (basic sketches of the skeletal framework of a website) evaluation could identify the same issues as a completed app, but the cost to adjust a wireframe is must less than having to reconfigure a completed application. For some products, multiple usability studies might be required to be run at different points along the product development path to evaluate different aspects of the product.

How to use them

The format to conduct a usability study should be well structured, and planned well in advance to allow for piloting the study. This ensures you are able to capture the information that is most important for you to progress in the development of the product. A set of tasks you wish the participant to complete should be constructed, as well as a task scenario to accompany each task. The tasks could be independent of each other or sequential in nature; either way, the tasks used should be those that have been identified as being most common, critical or important, or having been previously conducted incorrectly.

Understanding the users’ thought processes during the usability test is paramount in identifying points of frustration, misinterpretation, and opportunities for improvement. To achieve this a ‘Think Aloud’ method is commonly adopted. This method invites the users to provide a continuous verbalisation of their thought processes, detailing what they are thinking, and describing difficulties they may be having when performing the task (e.g., not knowing where to click to submit a form on a website). Thing Aloud can be conducted concurrently with the task or retrospectively. A retrospective Think Aloud may be best suited in particular healthcare settings when observing the user perform an action in a setting where they cannot verbalise their thoughts at the time; such as interacting with a patient or performing a task that requires a significant amount of cognitive effort. In these instances, having the user conduct a retrospective think aloud while watching a video playback can deliver similar results. However, the shorter the time between the recording and the playback the higher the accuracy. Alongside Think Aloud, ‘Probing’ is another technique that can be carried out, where the researcher may ask follow-up questions when they identify something they have not seen before or based on something the participant expresses that they would like to explore further. Again, probing can occur either concurrently with the task or retrospectively.

The method by which you wish to capture the users’ actions should also be considered, these could include: sound recording, screen recording, participant recording, action tracking, gesture tracking, gaze tracking, or any combination of these.

A usability test should consist of the following basic plan:

• Welcome the participant and explain the purpose of the usability study.
• Explain to the user how you are going to capture their thought processes (Think Aloud and/or probes, and either concurrent or retrospectively).
• Remind the participant that you are testing the product not them.
• Give them a task scenario and have them start the task (take good notes during the study as it is easier to take notes, rather than trying to recall the study at a later time).
• Continue the tasks until either they are all completed, or the allotted time has ended, as this can also indicate the difficulty of the tasks.
• At the end of the session you may want to provide an exit survey (such as a System Usability Scale (SUS)), to identify users’ initial perceptions of the device, thank the participant and provide any agreed incentive.

It is important when conducting a usability study that as a researcher you remain neutral, that is, do not provide opinions during the study and try to avoid providing assistance to the participant when they are getting frustrated. If the participant decides to give up or asks for assistance you can either end the scenario or provide minimum assistance. The reason you may wish to provide assistance is when the following task scenario is dependent on the participant completing the current task scenario or they are only on step 2 of 10 and you want to assess whether all steps are as hard as this particular step.

Pros

Good insight into product usability, engage with representative users, ability to identify issues early on in the product development.

Cons

Time consuming, difficult to recruit representative participants, difficult to engage participants in the study.

Points to ponder

• Do you have the right type of participants?
• Are the tasks the participants are carrying out the most critical?
• How are you going to analyse this information?
• Have you tested your equipment to know it works correctly and you are capturing the information that is important to you?