Research Methodologies
      Back to home
      1. Knowledge Base
      2. Research Methodologies

      Setting Up and Analyzing MaxDiff Experiments in SightX

      This is an overview of how to use MaxDiff questions and experiments in SightX surverys.

      What's on this page:

      What is MaxDiff Analysis?

      MaxDiff (also called Best-Worst Scaling) is an established market research method and academic mathematical theory about how people make choices. It assumes that given a set of items, respondents evaluate all possible pairs of those items and choose the pair that reflects the maximum difference in preference or importance.

      For example, consider a set in which a respondent evaluates four snack items: cookies, potato chips, tortilla chips, and popcorn.

      If the respondent says that cookies are their most favorite and tortilla chips are their least favorite, these two responses inform us on five of six possible implied paired comparisons:

      1. Cookies > potato chips
      2. Cookies > popcorn
      3. Cookies > tortilla chips
      4. Potato chips > tortilla chips
      5. Popcorn > tortilla chips

      The only paired comparison that cannot be inferred is potato chips vs popcorn. In this way, maxDiff questions are a robust way to get information not just on what respondents prefer/like/want/are likely to do, but also on what they value least/don't want/aren't likely to do.

      When to Use MaxDiff Analysis

      MaxDiff is useful whenever you want to measure the relative importance of product or service features, attributes, or respondent behaviors. Using the analysis output, you can evaluate which attributes are most and least important to inform trade-offs and areas of focus when developing a product or service.

      There are two types of MaxDiff in SightX: basic and experiment. MaxDiff basic presents the respondent with one set of attributes, while the experiment presents the respondent with multiple sets of attributes, each with different combinations.

      MaxDiff Basic

      Below is an example of a MaxDiff basic question in a survey. The basic question is best for smaller attribute sets so that you do not overwhelm the respondent with too many options, because when there is a longer list, respondents cannot reliably pick which is their favorite/best and which is the opposite. If you have eight or more attributes, you should use a MaxDiff experiment.

      MaxDiff Experiment

      Below is a video example of a MaxDiff experiment in a survey. As you can see, eight attributes are presented in combinations of four to the respondent, and the respondent selects the least important and most important out of each set.

       

      Setting Up a MaxDiff in SightX

      From the Create page, click "+ Add Item" and select either Max Diff basic from the Questions category, or Max Diff Experiment from the Methodologies category.

      MaxDiff Basic

      To set up a basic MaxDiff question, enter your question text as you would with any other question. In the Column Headers dropdown, select the comparison options you want to use for your question. Then, enter the attributes you want to compare in the rows.

      MaxDiff Experiment

      To set up a MaxDiff experiment, enter your question text as you would with any other question. In the Column Headers dropdown, select the comparison options you want to use for your question. Then, enter the attributes you want to compare in the rows.

      The next inputs are the Number of Sets and Set Size.

      • Number of sets = the number of combinations that each respondent will be shown
      • Set size = how many attributes will be shown in each set

      In this example, each respondent will see three combinations of attributes, so they will answer three times which are least and most important. Each combination will consist of four attributes.

      Helpful Tip: We recommend limiting the number of sets to 10 and to limit the set size to six or fewer attributes to prevent respondent fatigue.

      Sample Size

      When fielding a survey with a MaxDiff experiment, it's important to sample enough respondents to ensure enough data for the analysis to be accurate. You can use the formula below to calculate an appropriate sample size, or use our MaxDiff Sample Size Calculator.

      Minimum Sample Size = 500 * (number of attributes / (number of sets * set size)) 
      Recommended Sample Size = 1000 * (number of attributes / (number of sets * set size))

      Analyzing MaxDiff in SightX

      The analysis output of a MaxDiff experiment can be shown by Appeal Count, Total Count, or Utility Ranking.

       

      Appeal Count

      When the chart is configured to show the Appeal Count, it considers all of the sets in which a statement was selected as the most important or least important, and then shows the percentage of times it was selected as the most important, the percentage of times it was selected as the least important, and the net score. The appeal count does not consider the times when a statement was shown in a set, but the respondent did not pick it as either the most or least.

      The graph above tells us that:

      1. Out of the times that "No crashes due to driver error" was selected as the most or least, it was selected as the most important reason 82% of the time, and the least important reason 18% of the time. This means that "No crashes due to driver error" is typically a driving reason for respondents to consider purchasing a self-driving vehicle.
      2. Out of the times that "Ability to sleep while in transit" was selected as the most or least, it was selected as the most important reason 19% of the time, and the least important reason 81% of the time. This means that "Ability to sleep while in transit" is typically not a reason that is important when respondents are considering purchasing a self-driving car.

      Total Count

      The Total Count view of the MaxDiff output differs from the Appeal Count because it takes into account the times when respondents saw an option in a set but did not select it. The appeal count shows the percentage of times that an option was selected as most important or least important, out of the times it was overall shown to respondents in the experiment.

      The graph above tells us that:

      1. Out of the times that respondents saw the statement "No crashes due to driver error", they selected it as the most important reason 49% of the time, and the least important reason only 10% of the time. This is significant, because it means that roughly half the time this option was presented, it was the most important to respondents.
      2. Out of the times that respondents saw the statement "Ability to sleep while in transit", they selected it as the most important reason only 10% of the time, and the least important reason 43% of the time. This is significant, because it means that roughly half the time this option was presented, it was the least important reason to respondents.

      Utility Ranking

      The Utility Ranking view of the MaxDiff experiment output ranks each option by its relative utility to respondents, based on the comparison between how often an option was ranked as most important vs least important. Here, utility shows the satisfaction or benefit to the respondent derived from that attribute.

      The graph above can be interpreted such that customers will receive some benefits and satisfaction from all attributes because all utility rankings are above 0%. Respondents will receive the least benefit or satisfaction from a snack that 'Comes in a single-serving package', and will receive the most benefit or satisfaction from a snack product when it 'Pairs well with dips or spreads', is 'Packaged without plastic', and is 'Satisfying to eat without a dip or spread'.