Recently, my colleague Ryan Lekivetz wrote about our trip to Discovery Summit Europe in Brussels and our plan to test whether Belgian chocolate was really better-tasting than US chocolate. Ryan has blogged in detail about the constraints of designing the study, as well as the factors involved. In this blog entry, I’ll describe how we collected the data and analyzed the results.
Sensory analysis is a big topic with a well-developed literature (MacFie & Thomson 1994 and Næs T, Brockhoff PB, Tomic O. (2010) are good references to learn about the topic). We just wanted to answer one question: In a blind taste test, would people prefer the Belgian chocolate we’d carried across the Atlantic Ocean — or the US chocolate we could purchase at the corner grocery? Given the dollar/euro exchange rate and the limited space in my suitcase, this was an important question to answer before Discovery Summit Europe 2016!
Ryan’s designed experiment not only balanced on our two hypothesized effects (origin and cacao content), but it also randomized the question order. Study participants get tired, especially in taste tests, so we needed to make sure people were seeing their chocolates in the random orders assigned without giving away which chocolates they were tasting. I was also tasked with finding a US chocolate that was of comparable quality to our Belgian chocolate. More on that later.
Clean data collection requires preparation.
I therefore spent an evening at my kitchen table marking paper plates with choice set numbers, Survey IDs and a “Start Here” arrow (see above). Each of the survey subjects would get a plate with five side-by-side pairs of chocolates. There are 10 pieces of chocolate on each plate, and 25 plates in all (250 pieces total).
Sheila Loring, industriously sorting chocolates.
Each side-by-side pair represents a choice set in the experiment, and each subject had to choose which chocolate of each pair (choice set) he or she preferred.
Setting up those 25 pieces of chocolate was not an easy task. Many thanks to our documentation specialist Sheila Loring and JMP tester Bryan Shepherd for their assistance. (Strangely, they wouldn’t accept extra chocolate as a thank you.)
Once we had the paper plates loaded, it was time to run the surveys. We did our surveys in two groups so we didn’t have to wrangle 25 participants at once; at the beginning of each session, we made sure the participants knew how to sample their chocolate and mark their plates.
If you try this at home, I strongly recommend running the surveys just after lunch. It’s hard to make your test subjects listen to orientation when there’s delicious chocolate sitting right in front of them.
Two of our test subjects, pondering the mysteries of chocolate.
Once we had the data collected, I entered it into JMP. My first job was to make sure we’d collected the data correctly, and for that I used the Categorical platform. Ryan’s design was balanced, but we’d also collected respondent information (chocolate preference and JMP division) as part of our survey information. I also wanted to make sure my data entry was correct.
Note: This was run on my choice response data, which has 10 rows per participant, so there are actually five people who don’t prefer dark chocolate, not 50.
I did find some data entry errors using Categorical. Sometime during the data entry process, I got tired of typing “Development” over and over and started typing “Dev” instead. I also had to change how I measured chocolate preference because instead of checking “Milk” or “Dark,” two of my survey respondents got industrious and wrote in “Neither.” Figure 1 shows the data after I used Cols->Utilities->Recode to clean it up.
Once I my data was clean, it was time to use the Choice platform. Analyze->Consumer Research->Choice took me to the dialog, where I was able to specify how the data were collected and enter the effects.
In Figure 2, you can see that the Grouping columns are: Participant ID, Survey ID, and Choice Set ID. This tells JMP which column IDs uniquely identify a single choice set for a given participant. See the Choice documentation for more information on how to specify grouping columns for a single-table survey.
Once I had my grouping columns identified, it was time to specify the effects. Figure 3 shows the effects I first entered into the model. Notice that Chocolate Preference and JMP Division only appear as interaction effects. Choice models always compare the Choice a subject made to all of the choices he/she was shown in a given choice set, which means you can’t estimate a main effect (fixed) for something that is subject-specific (like whether a person prefers dark chocolate). You can only estimate how it affects subjects’ preferences for product-level factors (like percent cacao in a piece of plain chocolate).
For this analysis, I was really glad the Choice model dialog stays open even after you click the “Run” button. Using the AIC (lower is better) and the Likelihood Ratio tests, I was able to whittle my model effects down fairly quickly. Figure 4 shows my final model. You can see that origin isn’t statistically significant at the 95% level. I left it in since it was our variable of interest, and I wanted to discuss it in context with the other effects, but I’m pretty sure I had my answer by Figure 4.
Figure 5 illustrates the results using the Probability Profiler:
In this case, my comparison group is the Belgian chocolate with 41% cacao, and I make sure the subject effects are consistent (so dark-chocolate lovers are compared to other dark-chocolate lovers).
As you might have guessed, dark-chocolate lovers prefer dark chocolate (as you can see from the fact that the line goes up, and then up again as cacao content increases). They also appear to have a slight preference for Belgian chocolates over US. When I move the toggle for both the base and comparison products preferences to “Not Dark,” the slight preference for Belgian chocolate remains, but you can see a sharp drop in “Probability of Purchase” between 41% and 60% cacao. (Figure 6).
In fact, the Probability Profiler shows that if you presented our five test subjects who did not express a preference for dark chocolate with two pieces of chocolate that only differed by cacao (one 41% cacao and the other 60% cacao), the our model predicts only a 1% chance that they would choose the dark chocolate. Percentage of cacao is the most important factor in predicting which one people will prefer.
So the winner is...
If we compare US vs. Belgian, keeping the cacao content and chocolate preferences constant, we see that there’s a much smaller difference in the predicted probabilities. If we have two pieces of chocolate with the same cacao content, one from the US and one from Belgium, and asked someone to choose between them, the model predicts a 39% probability that they would choose the US chocolate, and a 61% probability that they would choose the Belgian chocolate. One could argue there's a slight preference for Belgian chocolate.
I did try some other variations, including an analysis without the five milk-chocolate lovers. Origin was statistically significant in some of the competing models, but the effect stayed the same: 39% probability US vs. 61% Belgium. The data is available on the JMP User Community if you want to try some analyses of your own.
Now, a study of just 25 participants is pretty small, and if I were working with a customer, I’d caution against making a huge decision based on my analysis. However, when it comes to deciding whether I’m going to stuff my suitcase with heavy, melty, space-consuming chocolate on my next trip to Europe, I think I have my answer.
The most important thing is to make sure I bring milk chocolate for my friends who like milk chocolate and dark chocolate for my friends who like dark chocolate.
Controlling for that factor, Belgian chocolate might be slightly better (though I can't prove it's significantly better), but the effect is small.