‘In the Wild’ testing
Key Points
- Goes beyond user experience testing, as it evaluates the product being used within its designed context.
- Users are evaluated interacting with the product as they see fit, which can lead to identifying unforeseen issues by the development team.
What are they and Why use them
In the wild testing involves carrying out a Usability study and User Experience Evaluation when the product is being used in the context it was designed for by the intended users. It is important that in-field testing is carried out prior to moving towards the market as it allows the regulatory design requirements to be evaluated and to identify any unforeseen issues to be captured and their risk determined. This may include the product not being used as intended or being used in a manner it was not designed for, identify unknown constraints, and understand how the introduction of the product will modify existing workflows and task distribution. “Examples of the need for Human Factors” highlight issues that in-depth ‘in the wild’ studies should have been able to identify, thus allowing design changes to be conducted to improve the product’s ability to perform its task and improve user satisfaction and safety.
When to use them
In the wild testing is the last formal test carried out by researchers before moving to mass production and can provide a summative evaluation of the product. During this testing, unforeseen challenges may become apparent that could not be picked up in usability and user experience evaluations conducted in non-contextual environments. Therefore, carrying out these studies again but being used in the environment they were developed to be used in is essential. The duration of testing also needs to be taken into consideration to ensure that enough data is captured to ensure a realistic evaluation of use is captured. For example, when you have designed a new dialysis machine than this product may only be used every couple of days, so, the time frame may be dependent on having the user perform dialysis a set number of times. However, if the product being developed helps take common patient bedside readings, such as temperature, heart rate, oxygen saturation levels, etc., then an evaluation period of a week may be sufficient.
How to use them
In the wild testing involved reconducting usability and user experience evaluations, with the main difference being the location and the users. To ensure that all possible outcomes are understood, evaluation of the product should be conducted by as many participants as possible at varying locations. For example, a device to be used by neurosurgeons should ideally be tested at a number of hospitals by both senior and junior surgeons. Similarly, a product developed for in-home use should be evaluated by different households and trying to capture a wide range of users.
Pros
Understand how the product works in its true environment, identify unknown constraints and exactly how the introduction of the product modifies existing workflows and task distribution.
Cons
Difficult to manage data collection and (depending on the product) may be difficult to conduct evaluation studies. The studies will also generate a large data set, which will need to be analysed and assessed.
Points to ponder
- Does the product impede existing practices?
- How many locations and users are required to properly evaluate the product?
- Is the product being used as intended?
