The Great Virtual Cookie Experiment: The results
Created:
At the last social event of our online analytics conference JMP Discovery Summit Europe 2021, attendees voted on a series of pairs of photos of cookies. Each pair of photos were taken by volunteers according to instructions based on a designed experiment. For more details about the experiment, there is a separate blog post outlining it.
The attendees needed to pick which of the two pictures tempted them more from each set, picture A or picture B.
While not immediately known to the attendees, the factors being changed by our photographers were:
- Beverage: Milk / Tea / Coffee
- Type of Plate: Solid / Pattern
- Setting: Inside / Outside
- Number of Cookies on Plate: 2 / 6
- Image: Color / Black & White
- Use a Prop: Yes / No
Many thanks to Clay Barker for helping collect data during the social and prepare the analysis. He set up the data set to be used with the Bradley-Terry model as has been seen in a couple of previous posts (here and here). Full disclosure: Since the data was being collected real-time and Clay had percentages available, the percentage of each pair of pictures was entered. So it pretends we had 100 respondents, when we had between 70 and 80 at any given time. Modeling this as the probability that picture A would be selected, during the social Clay showed us the main effects model in JMP:
What this tells us is that based on the respondents, six cookies are better than two (which came up in the discussion), color is preferable to black and white, and there might be a small effect by using a prop.
After the social, considering interactions and going through some model refinements, we ended up with this model:
Still much the same story – six cookies and a color image are the main drivers, but milk/coffee/tea and the outside setting can be, depending on what you are comparing it to. The interactions factor in what the paired picture has, so it can be a bit more difficult to see from here. If you want to dig more into the details, I have attached the original data table and the one set up for the Bradley-Terry model.
Clay was kind enough to set up the Profiler as well so that you can compare the probability of choosing picture A given settings as defined by pictures A and B. If you want to explore the model, the interactive Profiler is available on JMP Public.
According to our model, these two pictures should look tempting. Let me know in the comments if those plates of cookies indeed look good to you (or not so good).
Final thoughts
It would have been interesting to collect some demographic information to go along with the results, but because the event was virtual, we wanted to keep it as straightforward as possible.
If we ever contemplate an experiment like this again, I would change how we handle the prop, which was left very open-ended for this experiment. Being much more specific about the type of prop would give a better idea of whether using a prop makes a difference.
Attached to this post are a couple of versions of the data for analysis and a set of slides with the pictures, if you are interested.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.