A MaxDiff Bake-Off Study: Finding the Most Preferred Among Dozens of Products (2021-US-45MP-804)

[#1]

Invited Paper On Demand Winner

Level: Intermediate

Rob Reul, Managing Director, Isometric Solutions

Amy Anderson, Principal, AA Insights, LLC

Bradley Jones, JMP Distinguished Research Fellow, SAS

A MaxDiff study is a way of ordering consumers’ preferences for a group of similar currently marketed products. As the number of products to be compared increases, the cognitive burden on respondents can rise to the point that few will complete the survey, which could lead to potential bias. In our case study, we were tasked with ordering preferences for three dozen (36) products. We addressed the problem by first dividing the 36 products into four mutually exclusive groups of nine products and did separate MaxDiff studies on each. The “bake-off” consisted of a new MaxDiff study pitting the most preferred products from each of the four groups.

We present the analytical results of the study using JMP. We also show the economic savings resulting from streamlining the original offering of 36 products to a substantially smaller group of preferred products.

Auto-generated transcript...

Speaker	Transcript
Jordan Hiller	Go.
Rob reul	the JMP Discovery conference, where we're going to be talking about portfolio and product optimization.
	This screen is a foreshadowing of our entire talk today, and it is solving this pRoblem for what this consumer might go to look for either, online or in a department store or many other places gift cards are sold.
	Amy Anderson, their chief analytics consultants; and myself, Rob Reul, the managing director of Isometric Solutions.
	So Amy is going to tell you a little bit about Blackhawk.
Amy Anderson	So I was introduced to Blackhawk a little over a year ago, and it was a colleague of mine who is actually in charge of all gift cards at Subway who was lured to Blackhawk to help them with their card portfolio.
	Most people have not heard of Blackhawk. They're behind the scenes, but they're actually the world's largest privately held gift card
	company. They position themselves as a payments brand because they do digital gift, phone, debit, incentive cards, online and through a global network of retailers.
	If you think about what Rob just showed you a couple slides ago, that's Target. You know, you've pRobably been to the grocery store.
	You see the cards that you have options that, you know, as you're checking out. That is the main focus of their business; it's 97%. They do also have an online presence that they're starting to grow through...
	through giftcards.com and other web entities. So when Blackhawk came to me, and they had a challenge. So when you think of gift cards
	and that whole gift card mall that was on the first slide, there's only so many pegs,
	and think of each peg as rental property. So you've got to pay in order to be on that peg. So as a company that is producing gift cards, they need to understand which cards are going to be successful and deserve to be on that peg.
	Blackhawk had always,
	kind of, you know, guesssed at which cards should go out there, and what would happen is they'd wait 12-18 months to see if the financials came in.
	And that's a lot of time to waste, so they came to us and asked, you know, is there a better approach to figuring out which cards we should put on those pegs to appeal to the most customers?
	And that's where I brought Rob in and I'm going to pass it back to Rob to talk about how we approach this.
Rob reul	Yes, and and...
Amy Anderson	Before we go there, I'm sorry, could I
	talk about the try vs the test?
Rob reul	Well, you better, because that was what your construct was based on.
Amy Anderson	So anyway, one of the things that Rob and I've talked about, and we laugh and and a lot of our marketing colleagues laugh as well.
	Blackhawk is famous for doing what we call tries. So a try is very different than a test, and when you think about it, it's...
	you know, they put something out there and they'd see if it worked. And we are trying to...we're trying...we are working with them to approach things with more of a test in mind, so looking at, you know,
	the trade offs. And with this test that we've designed for them, it's really all around choice, because when you try something, and you wait 18 months, it's too late to find out that it was a failure.
Rob reul	Indeed. And so when Amy talked about Blackhawk guessing, they actually got quite good at guessing, right, but the the size of their marketplace and the number of cards
	became quite, quite large and you're going to soon see that. So in terms of doing a test or an experiment, we baked up the following.
	So they had...I think they had 113 cards that were vying for those pegs and the challenge, of course, was to design the trial
	or the experiment that would allow us to do that. And this is a snapshot of what that ultimately became. I'm going to walk into how we got...we got to there
	in a second, but you're going to see a couple key things here, something that we're going to talk up as a tournament design of a designed experiment.
	Attributions first would go to Dr Chris Nachtsheim and then working closely with Dr Brad Jones on, basically, first the concept and how do you actually then do that.
	And then it became Max Diff experiments, which now, of course, is a...is a crucial platform for me in JMP that I use all the time.
	And we really put it to the test here by doing, in this specific study, five different Max Diff studies that...four that fed each other and one to get the finals, and we'll walk you
	through how we got to there. So first let's talk a little bit about cards. So you saw that wall, 113 or so plus competitors' cards. That was Target.
	When we looked at the Blackhawk scenario, we couldn't conceivably build an experimental design that could be fielded the consumers
	and expect them to complete that in a reasonable number of time. And by that I mean nine minutes...nine to 12 minutes.
	I'd done a bunch of work again with Dr Nachtsheim on modeling fatigue and choice experiments and
	after about 12 minutes, people just start answering questions and you have internal inconsistencies, people answering all in the same place or people not finishing the survey.
	And of course, we didn't want that at all, so we had to winnow it down to the set that was most material and that first effort resulted in
	36 cards. So there's some, there's more, and there's the rest, right. So that was sort of our set of 36 final cards.
	Now the conventional way most researchers, without deep survey analytics such as Amy and I are bringing, would solve such a problem, would look like this, and this is actually something I've talked about before
	in a JMP conference. But you see the classic rating question, where each of the rows is one of the things we're evaluating and, in this case, it would be 36 rows tall, right.
	And you have a how likely would you be to buy, or which card is your favorite card, and after a while, you can see, you can imagine, how useless the data that you would collect would be. You'd get
	very poor separation in being...in terms of being able to analyze it, and so ratings were certainly were out.
	The next approach typically you'd see in our field is somebody...well okay, let's rank them, because now they have to think of them not independently but interdependently.
	And that's a better way, but if you have a long list like we do, you have a completely exacerbated set of challenges for respondants, causing massive cumulative cognitive burden.
	And it's basically untenable. And so of course, we have the fix, so that's the build up to the method, this is Loviere's
	Best-Worst Scaling. And Max Diff is how we know it, and it gives us a magical statistic, known of as the utility. How much is one card worth
	over another card, right? And so that's what ultimately we're after. So here's our experiment. This is what we were set up to do, and I'm just going to break the JMP...
	the discussion there. I'm going to go into my
	JMP designer and show the experiment that we actually did, and it is here.
	I can get it.
	And we'll make that design.
	A couple things I want to highlight about this design...(if it comes over for me. Hang on a second.)
	Can you see it, Amy?
	All right, good.
	So, again we had 36 cards, so to break this into manageable pieces,
	we ended up using block sizes of nine...or five with nine different groupings, which had us then serve up 18 choices in a different experiment, and we had
	a design that looked like this.
	And then we programmed it, and we have an incident matrix that actually is a beautiful thing.
	So in in in our world, it's really important to make sure you compare every card against every other card.
	And don't make assumptions about one card beating another card, because it'd beat cards that one of the other cards beat or didn't beat, right.
	And what we have with this balanced and incomplete block design is a...is a beautiful efficient experiment, where every card was tested against every other card exactly twice.
	So we program this experiment, and I want to show you now what this looks like once it's programmed, and this is kind of a beautiful thing as well.
	I'm going to go back to
	my experiment here.
	So we took the
	designed experiment, we built it into an interface, and in the middle of the survey, about four minutes in, all respondents were
	asked to...where did it go? Hang on a second. Sorry about that.
	Try that again. Sorry about this.
	They were introduced to the experiment
	and then we showed them
	the randomized groupings or choice sets, where they were asked to evaluate each card. So if I like this one the most
	and I like this one the least.
	And we had them do this several times,
	based on this design, actually 18 times, to be exact.
	And each time we prompted them along the way to keep them in the game, so to speak, and once we
	were able to get people through all 18,
	we had a set. And so we broke the experiment into nine different...
	four different groupings of nine each and we had an aggregate set of respondents and then we took the winners with the top utility scores and put them in a final heat. But we did it again, and so we ran them through the finals.
	And...
	let me get back here.
	It's a little kludgied, I apologize.
	There was the experiment.
	And then we analyzed it based on the work of Danny McFadden and the analytical platform that's in JMP, right.
	Again, the key here, he comments at the bottom of his text about the fundamental concern
	understanding human choice behavior and to do so in an environment where you don't have confounding influences that you can't interpret because you don't know what was being discussed or what a respondent was influenced by.
	So this is a look at the marginal utilities, right. So this is the first cut at what came out of the experiment once we analyzed it. I'm not showing you all the exact cards, but you get the idea, positive and
	negative, those that were not statistically significant in the middle. So we had a big array like this of 36 cards.
Amy Anderson	Can I mentioned something here, Rob, that was super important? So as Rob mentioned earlier, a lot of survey research people are just asking, do you like this? and rank it from one to five, and that's how Blackhawk has approached all of their research. So when we showed this to
	our product marketing people, they were shocked and they didn't understand. They're like, wait a second, why is happy dad down there in the red, because, I mean, that had a top two box.
	Well, the difference is because if you ask them, do you like this, without giving them the choice, they may like it, but what if there's something better? And so this was just
	game changing for them, because they've never looked at their card portfolio that way or approached research this way.
	So keep going, Rob. So I just wanted to clarify that, because it was, I mean for them they just...they never thought about choice. It's, like,
	well, obvious that you need to with all those choices out there.
Rob reul	And they also have a wall; there's there's an awful lot of choices, right. Yep.
	So we had this data, but but Amy was was right to remind me that was only part of the equation here, and I'd like her to speak again.
Amy Anderson	Yes, yeah thanks, Rob. So as I think I mentioned it earlier, Blackhawk is very financially focused, as most businesses are, I mean, you need to make sure you're making money.
	However, you know, making money is is the result, you know, that's after the fact. And so that's why we needed to look at this research. And so the affinity index, which is great, understanding do you like it? But you also have to understand
	okay, that's great, I like this better than that, but
	are you going to buy it? And if you're going to buy it, how much are you going to load.
	And so, by having this as part of the research, it's helping them bridge that gap between what consumers are telling them and what they're gonna do.
	And over time, this is helping us develop a scorecard for the client, so that they can get consumer feedback early on, and eventually estimate
	what's going to happen to their financials if they act on that information, and that's why it's so important to have all three of these metrics as part of the study.
Rob reul	So, by combining them, we were able to create a predictive model of their book of business looking forward.
	And what Amy was able to set up for them was a way then to go back and look at it, basically saying, if you...
	if you advance these cards in the market, they look to generate this kind of revenue in six months or 12 months time.
	And the next screen shows how we ended up being able to lay that out for the client. And so we we took the
	affinity scores, and if you can see each each card here that was ranked based on the basically the utility, and then we have the likelihood that it would be purchased.
	And we have the load value, right, because we asked everybody okay, that was your favorite,
	how much would you...how likely would you be to buy the card and how much would you spend on it? And then we could create this adjusted value that she talked about.
	And then we could scale rank all the cards relative to each other, along this axis, which was basically the utility against the likelihood of load. And we could we could create some separation and we can find for them
	the very best cards and, in this case, clearly this Visa card was was a predominant winner.
	And we were ultimately able to understand why and we'll talk about that in just a second. It has to do with the broad appeal and the number of brands that the card can be used on, right.
	But this was the way we were able to create that separation. So that was the first main piece of work that we were able to do. I'm going to move now to the next level of analysis and Amy is going to talk to you a little bit about the anatomy of the gift card.
Amy Anderson	So um so we just looked at the portfolio optimization, so we had the rank ordering of which cards are going to be more or less successful.
	And...but then the second part is, there are some of those cards that maybe are in the middle, that they don't need to throw away, but they they need to think about how they can optimize them. Now
	when you think of gift cards, I mean, until we got to this business I didn't really think about all of the components of a gift card.
	And with this specific portfolio, which they call their happy cards, there are a lot of things on the card that drive preference.
	You know, it could be the card load, so how much could you buy the gift card and gift for somebody? It could be the condition. So what do I need to do in order to use the cards.
	Personalization. Are you able to personalize it like to mom or my daughter, etc. We have the brands on the card, and then we have the card theme or the occasion.
	And so understanding which of these elements are important to driving preference was the next step, in terms of understanding how...where to focus to optimize the cards.
	So um so we had basically limitless number of themes, denominations, of personalization options.
	And the way that Blackhawk typically would do what they call their card curation, which was coming up with those cards, is they would sit in a room...in a conference room, pre-COVID,
	and literally go through all the past financials and say hmm, I think this card performed well because it had this brand.
	Oh, I want to maybe take this brand because we haven't put it on a card. And it was very, very,
	you know, they have a great business background, but somewhat subjective in terms of the approach to figuring out how to get those cards created.
	So this study allowed us to understand what was driving that preference (and I'm not going to give away the prize, right now, because I want Rob to walk through it)
	but it's developed quite an amazing tool for the client.
Rob reul	It worked, right.
Amy Anderson	It works really well.
Rob reul	the choice platform. And
	what we're looking for here is a discrete choice experiment, where we can actually serve up and trade off those things that we know are critical, based on the anatomy that you just saw.
	And that took shape in the experiment that's shown on the right side.
	That's a live image of the screen, obviously going back and forth between JMP in explorer and the deck is not optimal for you as a viewer, so
	I'm just going to talk about it here. So we had four main, sort of, themes or attributes, and within attributes we had several different levels. The main ones tested, based on the anatomy, were what's the theme that compels people to buy the card?
	Who are they giving the gift to, because that turns out to be a big factor. The sponsors or brands on the card, and conditions of use. So sometimes there is a expiration date, sometimes you have to register the card online.
	Right.
	Sometimes you can only redeem a card online, so the idea was basically to figure out which of those was the most palatable, right, sort of as a cost of entry.
	And turns out, as Amy was was hinting at, we got a very strong model as a result, but, for one...one of the variables kind of fell out. But you can clearly see...and Amy was going to...
	was going to tip her hand and tell you that 57% of the cards preference is all based on what brands are on the card.
	We knew brands are important, but we didn't know it was that important, right, but the other factors are important as well. And you can see the profiler on the right,
	where that first variable kind of washes out and it's not statistically significant, but clearly we had some nice separation between the different attributes on this...in the study. And these were all nominal attributes, which
	makes it even more interesting, right, because you have to be careful when you're moving between. So that was kind of the the main...the main output there.
	And we had a...we had a conundrum though, right, because okay, they had 253 brands on a card.
	Now, with the first study we showed, we had 36 cards and we used five Max Diff experiments to figure that out, because we could get humans to process that when asked about it.
	Here we had a discrete choice experiment. I think there were 12 screens, each screen had three choices, so not a trivial experiment for an individual to think through. But how do we solve for
	the number of brands and how the brands work? And so that begat this whole notion of a designed exercise, which was not a designed experiment, but, but much more better...much more deliberate than just a try. So what we actually ended up creating was this notion of a dream card designer.
	And after we put people through the discrete choice experiment and constrained their choices to the ones shown,
	we asked them to choose and build their own card to see what they would like. And the same components come alive, based on that anatomy.
	So this took us in a whole different direction. Again, what we're solving for is, well, what brands, should they entertain on their cards?
	And it turns out that the amount of compute to solve the feeling of this study from 2,000 respondents took about three days to run.
	It was an enormous amount of combinations when they could choose between any of the 253 brands, just one or all the way up to nine.
	In addition to who the card was to be given to, whether they wanted a denomination and how big that denomination would be. You can imagine, it was it was massive, right.
	But what it was able to produce was this sort of amazing output that I'll let Amy talk about.
Amy Anderson	So this information, along with the output from the other study, we took and we put it into what we call a card optimization tool.
	So that when the client would go and design new cards this tool, they could reference, and say, all right, well does it make sense to have brands that are across multiple brand categories?
	And what we found with this output is that consumers don't want a ton of variety, because if they want variety, they get a Visa card, and yet, they want more than just the single branded card. So it was something in the middle.
	Typically, when we had a brand, the consumer, when they would do this exercise, would stay within brand category. So if the brand they picked first was a retail brand,
	they'd finished designing their card primary...primarily with retail. Same went for QSR, same went with digital, big box, etc.
	We had a few cases where they might cross brands and it would be instances where it would be retail and then maybe restaurant.
	Think about like a mall experience, and so that was one where it was a little different. But we have brands here that we would call...
	and this resonated really well with the curation team...I call them datable brands or not dateable brands. And dateable brands are the ones that have a lot of brands that are paired together.
	And so that boded really well for creating these multi branded cards. But then you have brands that aren't dateable, like for 24 hour fitness and Alamo, Albertsons. It's like they just...they're on their own.
	And so what happens is when these brands come to Blackhawk and they want to play, they want to be on these cards because that's revenue for them,
	this is a tool, not only to design the cards, but to help them with the negotiations to say, look,
	if you want to play, you have to pay, because your brands aren't really pairing well with anybody. And you know what? Our consumers
	prefer other brands. And so having that leverage is just been amazing for them to actually have some insights in order to design cards, and then also to negotiate these brands, in terms of, you know, doing business with Blackhawk.
Rob reul	Exactly, exactly.
	So there's one more piece to this and that has to do with
	card
	construction. And we talk first about, okay, what cards are keepers and which ones aren't.
	One of the things I ought to have mentioned earlier was when we set up those those those preliminary rounds, those four rounds,
	we seated those rounds based on revenue from the cards from the prior year. And, of course, the revenue is the result of the attractives of the brands where the cards can be used.
	And so when you start thinking about the actual combinations of brands on cards, Amy was showing you the conditional probabilities,
	we were trying now to take that one step further. And you see the top 50 brands listed on the far right, and we're trying to create
	targetable consumer segments that would be likely to buy brands clustered on the same card. So this is that next layer of the analytic work that we're doing to try to understand, okay,
	is there a target that's big enough that we can reach gender, age, some other psychographic type of thing, right, that would allow us to refine the killer card by putting the right theme on there with the right brands.
	So this is sort of the state of the research. We, together, have just described almost a year and a half of solid research for Blackhawk.
	And I would tell you that 90% of all the heavy lifting was done with and done because of the power of JMP. So that's our talk.
	We'll be happy to answer any questions live. I believe this is going to be shown on the sixth of October. And Brad Jones, who was the technical resource for the tournament design
	and analysis, will be accompanying me for that. So for now, thank you for your interest and we wish you a good conference. Thank you. Bye bye.