When and how to run preference testing? | UX research #25

Preference testing is a popular UX research method. The key to its success lies in its good understanding and careful preparation. When and how to conduct preference testing? How do they differ from A/B testing? How can they improve the final product functionally and visually? Read on to find out!

When and how to run preference testing? – table of contents:

1. What are preference tests?
2. Qualitative vs. quantitative preference tests
3. When to conduct preference tests?
4. Preference testing vs A/B testing?
5. How to conduct preference tests?
6. Summary

What are preference tests?

Preference testing is a research method that involves sharing several (usually two to three) design options with test subjects and asking them about their preferences – which design they like better and why. Unlike other tests, preference testing focuses specifically on the visual aspects of products and design. This study can help you learn more about the user and their perception of the brand, and discover what feelings and emotions they have. Preference tests also allow you to evaluate a design in terms of its visual appeal, brand consistency, and overall credibility.

Preference testing proves handy in UX, as it gives researchers and designers direct insight into user tastes and information about what they think of different designs (visually). As a result, the method allows important decisions to be made early in the design process and saves the company from the unnecessary investment of time and energy in a design that is unlikely to appeal to the final user.

Preference testing aims to understand what visually appeals to the target user and why. The results of testing can be used at many stages of the design process, from general planning of color schemes or hierarchies on a page to specific decisions such as font and icon selection.

Qualitative vs. quantitative preference tests

We can distinguish between qualitative and quantitative preference tests. Qualitative ones usually take the form of an interview, during which we show the user different versions of the product and are then asked which one they like best. Qualitative tests also examine the user’s impressions and attitudes toward each of the designs shown, and allow them to answer not only which design they like best, but “why this one?”

Quantitative preference tests can take the form of a survey in which users select their preferred design and what attitude they have toward each. This allows feedback from a much larger survey group than qualitative tests, resulting in greater confidence that the results can be generalized to all users. Quantitative preference testing is appropriate when the design is relatively simple and does not contain numerous screenings, and when the company knows the reasons why users prefer one version more than another.

When to conduct preference tests?

You can run preference tests at various stages of the design process. However, they usually come early in the design phase to get initial feedback from users, even before the company invests time and money in the project. In this way, it can be determined which direction is more cost-effective and why.

Companies that are not building a product from scratch, but plan to redesign an existing solution, can conduct a preference test to check their design against the competition.

Preference testing vs A/B testing?

In the simplest terms – A/B testing is done later when the final design is almost ready and users can interact with it live, while preference testing is done early in the process – on a prototype, model, or even a sketch of the design. Preference testing is about understanding which designs a user prefers and why – before the product is finished.

Another key difference is that A/B tests rely on KPIs (Key Performance Indicators). They help determine how different variations affect user behavior and the achievement of business goals. As an example, a company that wants to increase the number of newsletter signups on its online store decides to A/B test the CTA button, examining which color of the item generates more conversions.

How to conduct preference tests?

1. Define research objectives

As with other methods, the most important thing is to define the objectives and research questions and write them down in a prominent place and present them to the test participants. Is the priority to understand which design variant users prefer? Is the goal to find out how they perceive each project individually?

It is also worth determining what kind of feedback the company wants – qualitative or quantitative. You should also make sure that all design options are functional.

2. Determine how to measure responses

Participants’ sharing of insights can vary depending on whether qualitative or quantitative tests were conducted. Here are some possibilities:

• Open-ended questions – this way participants can argue their choices, e.g. Which design do you prefer? Why do you like it? Was the information on this page clear and understandable to you? How easy was it for you to navigate this screen? What do you like about the design of this site?
• Closed list of adjectives – having specific adjectives at their disposal, e.g. classic, minimalist, elegant, participants will be able to assign them to the evaluated patients.
• Request to describe the variants in 3-5 words (adjectives).
• Numerical evaluation – will help determine which design best represents certain brand characteristics.

Participants in the preference test (as well as other research methods in UX) are mostly polite and usually refrain from direct criticism of the design, even if they are explicitly encouraged to be ruthlessly honest. For this reason, it is sometimes better to use indirect methods to find the real reasons for choosing one product over the others.

Gather test participants

According to the general principles of research with users, it is necessary to find test subjects who reflect the target customers as closely as possible. Depending on the stated goal, they can be current customers (if the company wants them to understand the context of use) or people unrelated to the product (if the study is to tip fresh insights, without bias).

Before recruiting, you need to determine the number of participants (20-30 respondents are recommended), the method of reaching them, and a possible incentive for participating in the study.

Conduct a preference test

Before starting the test, it’s a good idea to explain the entire process and expectations to participants. The test should not just show designs, but also ask follow-up questions. This will help the researcher find out why the participants chose a particular option, and how the prototype can be improved by considering their preferences.

Analyze the results of the preference tests

Conducting tests without analyzing the results does not bring much information to the company, so don’t skip this step. For qualitative testing, group similar responses and find the resulting patterns. With quantitative testing, it is worth extracting the most preferred answer. However, if the difference is not noticeable enough, retest on a revised draft.

Summary

Preference testing is worth conducting early in the design process, as it can help design based on user preferences rather than personal guesswork. Additionally, this solution is easier to implement and less costly than A/B testing.

If you like our content, join our busy bees community on Facebook, Twitter, LinkedIn, Instagram, YouTube, Pinterest, TikTok.