How can pilot testing improve UX research? | UX research #9

Today, we’ll explain what pilot testing is and how it relates to conducting UX research. You’ll also see how it can improve the entire UX process to deliver satisfying as well as reliable results.

Pilot testing – table of contents:

1. What is pilot testing?
2. What is pilot testing in UX research?
3. How can pilot testing improve UX research?
4. Summary

What is pilot testing?

Piloting, by definition, means the implementation of a preliminary project, which aims at identifying barriers and gathering valuable information necessary for further developments of the project on a larger scale. In the nutshell, it concerns preliminary, unofficial testing of a given product or solution as well as catching bugs – all to improve its performance before its actual release, or application. In this article, we will speak of “piloting” interchangeably with “pilot testing” and explain what piloting is in the context of UX research.

What is pilot testing in UX research?

Pilot tests in UX concern those research sessions that happen once or twice before the actual test or interview. This enables fine-tuning the usability of the survey to the respondent, which makes it easier and more efficient to conduct the actual survey at a later stage. It allows us to validate the content of the tasks – whether they are understandable and clear for the users, learn about the estimated time of the survey which allows us to plan it better, more realistically, and in case the test itself goes smoothly, we can gain additional data for research analysis.

In a pilot test, a session or two usually takes place before the main, scheduled test. This small number of sessions is typically sufficient to prepare for the real test and make sure everything is in order. The purpose of the trial is to assess and ensure that the full, proper study goes as smoothly as possible. There are several situations in which pilot tests prove particularly crucial:

• When the designer is new to the organization or team or has no experience in usability testing. This help in the first attempt with a test session, the results of which we can always omit from the official analysis in case of failure.
• Testing in an unfamiliar subject area. If you are not an expert in a particular area, pilot testing can help you prepare better.
• Conducting a remote, unmoderated test. Whenever instructions for a test do not come out directly from the researcher, yet get sent to test users who have to act independently. The instructions need pilot testing to reduce the possibility of misinterpretation. In the case of unmoderated tests, the participant has no opportunity to ask for clarification upon encountering a problem during the test.
• Conducting a quantitative survey. Typically, those concern large-scale projects that aim at retrieving statistically significant results. In such surveys, each session must follow the same pattern, thus they require a rigid scenario that undergoes robust evaluation.

How can pilot testing improve UX research?

Pilot tests make it possible to significantly improve the actual research with users – thus influencing the outcome of the project. One of the main benefits of pilots is that they are a so-called “dress rehearsal” before the study. This makes it possible to make sure that the moderator and the entire research team are prepared for all eventualities, to check that all the necessary materials are printed, the website works and is functional, the consent form is prepared and the (working) cameras are set up at the right angle. A dress rehearsal before the survey is a great way to check the readiness of the team.

Pilot studies also have another valuable advantage – they allow us to test tasks. If the user does not cope with the test (as we assumed) because of a poorly written task then we adversely affect the results of the study. Overly complicated, unclear task content takes time away from studying the interface, requires rewriting the task content and additional explanations, and distracts both the user and the facilitator. Often tasks that may seem clear and simple to use (researchers, designers) can be difficult and confusing from the user’s point of view. A pilot test allows you to identify such hotspots before the real study. This will allow you to calmly modify the content or scope of the tasks and ensure a smooth study – and better results!

Pilot testing is also important because of timing – it allows you to determine with better accuracy how long the actual testing will take and to better schedule meetings with test users. Estimating the time to complete a task based on your own experience can often miss the actual time needed. Therefore, conducting a pilot with one or two pilot users allows you to determine this with much greater accuracy.

The last advantage of pilot testing that we want to talk about is that if it goes smoothly, we gain data that we can use in the final research analysis. If the pilot session detects no major errors in the team’s preparations, our tester-user performs all the planned tasks and has no problem interpreting their content we gain valuable data that can stay. Though pilot test can provide us not only with valuable information about the course of the test and our preparation, but also about the product itself, the usability of the interface, and its functioning.

Summary

We hope we have been able to convince you that pilot testing is very important in UX research, and that it has many benefits for the project team, the course of further research, its results, and the final project effect in a broader sense. Finally, we have two tips for you on planning pilot studies:

• to get the most out of the pilot session, schedule it at least 1 day before the target study. Such a move will provide you and your team with sufficient time to make necessary adjustments before the next study
• when recruiting participants for the study, look for people who overlap with your user’s profile, so their feedback becomes more relevant (you will add this study to the final analysis of the results – as the test user will match the profile of your target one

We know that conducting pilot tests means additional time and effort for your team: you need to recruit additional participants, prepare materials several days in advance as well as plan then conduct an additional session (or even several). However, consider its benefits – by choosing to run pilot testing, you increase the chances that the final test will go smoothly, facilitating results that are valuable to your team and ultimately to your product.

If you like our content, join our busy bees community on Facebook, Twitter, LinkedIn, Instagram, YouTube, Pinterest, TikTok.