User Experience Evaluation

Key Points

• Understand users’ reaction to the products.
• Identify users' pleasure and satisfaction when using new products.

What are they and Why use them

User Experience (UX) evaluation may appear similar to a usability study; the difference is that a usability study focuses on effectiveness and efficiency whereas a UX evaluation evaluates the user perspective of the interaction with a product over time. From this we are able to understand the nuances of the product and the users’ satisfaction with it. Therefore, UX builds on usability.

When conducting a UX evaluation you are exploring the following questions; How enjoyable is the product to use? Is it helpful? Is it a pleasure to use? Is it engaging? Does it look good? Does it add value to the lives of the users? Through finding answers to these questions, you will be able to determine whether the product is something that users will want to use or further refinement in the design is required.

When to use them

As UX evaluation builds from usability, so it should be run after you have refined the product through usability testing. Unlike a usability study, a UX evaluation should be run over a set period of time or number of interactions with the product. Depending on the product being developed, these evaluations may take place in a lab setting (e.g., patient monitoring system) or participants may be able to test it at their homes (e.g., blood glucose measuring device). The method used to conduct the UX evaluation should reflect the location where users will be able to assess the product.

How to use them

The User Experience Honeycomb is an evaluation tool for UX, see Figure below. The honeycomb enables researchers to ensure that the fundamentals of UX are addressed, which are: useful, usable, desirable, findable, accessible, credible, and valuable (see UX Honeycomb for more details on each of these). To understand these needs you will need to adopt a longitudinal data collection technique, that is, something you can leave with your participants to capture their impressions of the product after using it. These investigations should be designed so that as many areas of the honeycomb can be considered and evaluated.

Longitudinal data collection techniques are designed to capture user’s thoughts, feelings, and actions when using the product. Typical modes of collecting this data include diary studies, card sorting, interviews, or surveys. Depending on the product, users, and timeframe, each of these modes have their pros and cons. Diary studies allow the users to record their own thoughts (either written or voice recording) and take photos (if able) of interactions they want to share. Card sorting activities vary, but typically include a set of predefined cards that a user can arrange in a prescribed way to provide insights. Interviews and surveys are classic means for researchers to gain insights. Within healthcare, set surveys may be the easiest method to incorporate; however, diary studies, although taking more effort to both prepare and for users to undertake, deliver better insights.

In short, longitudinal data study could be structured as:

• Use a method that best suits the users and the location where the evaluation can be conducted
• Ensure the method allows users to capture as many aspects of the UX honeycomb as possible
• Provide clear instructions to the users on how you wish them to engage with the method
• Let the users know the duration of the study, and when or how often you want them to evaluate the product
• Send gentle reminders to users to record their thoughts, feelings, and actions using the method you have provided
• At the end of the evaluation thank the user and provide any agreed incentive

Pros

Identify users’ desire to use the device; highlight key areas that need improvement.

Cons

Some users may give poor feedback, UX is subjective, and over time users may ‘go through the motions’ of providing information. Typically, a large data set and depending on the method used may require transcribing, which is time consuming.

Points to ponder

• Under what conditions will this product be evaluated?
• Does my method allow me to fully understand all facets of UX?
• How long does the evaluation need to run for?
• How many users do you need to understand the UX?