topic Re: Two alternatives in choice experiment in Discussions

Two alternatives in choice experiment

aus22 — Fri, 11 Oct 2024 02:20:42 GMT

Hi, I am designing an experiment with two discrete choice options, 2 factors each with 2 levels, and 1 with 6 levels. I have tried DOE consumer studies, choice design but unable to make this work. Any help will be greatly appreciated. Thanks.

Re: Two alternatives in choice experiment

Victor_G — Fri, 11 Oct 2024 08:28:15 GMT

Hi @aus22,

Welcome in the Community !

May I ask the context of your study ? Is it linked to consumer choice experiments ?

There are multiple ways to create a DoE with 3 discrete choice factors, using Choice Designs or Custom Designs platform, depending on your objectives and context.

The basic concept behind Choice design is to "pick a winner" between different profiles (factors treatments) used in several choice sets for different respondents. Analyzing the several selected options picked in different choice sets enable to evaluate the relative importance and significance of the attributes (factors) tested, and find the best level combinations in terms of utility score across all respondant responses. You can find an example here : Run the Choice Design and Analyze the Results (jmp.com) There may be more parameters to consider when using choice design, such as number of attributes that can change within a choice set, number of profiles per choice set, number of choice sets per survey, number of surveys and number of respondents per survey.
The Custom design is a more general (and versatile) platform, to identify significant terms and best levels based on an assumed model. It may however not be suitable for analyzing consumer/survey responses, the use of CHoice design may be more useful.

Depending on your objectives and needs, one platform may be more suitable than the other. Could you describe the context, objectives and needs of your experimental design ? If related to consumer choices, how many respondents and surveys are you expecting ?

I attached two designs as an example, one created with the choice design platform with basic settings and your factors descriptions, and another one created with the Custom Design platform. You can also read the documentation and look at the examples in the JMP Help section : Example of a Choice Design

Hope this first response will help you,

Re: Two alternatives in choice experiment

aus22 — Fri, 11 Oct 2024 10:13:22 GMT

Hi Victor, thank you for the help. The objective is to assess which one of the set of restaurant attributes are more important for consumers. We have three attributes in this study as I pointed out. I am trying to create a design that provides respondents two options (alternatives) to pick from, these will not be labelled. Each will have different levels of those attribute levels. Hope this clarifies the study? Could you help further please? I do see the design you shared and the Choice design seems to be the more appropriate one. In that design does the survey column provides the repeated runs for each respondent? Also could you also tell me how can I put constraints on one of the attributes to not vary? Thank you again.

Re: Two alternatives in choice experiment

aus22 — Fri, 11 Oct 2024 10:57:05 GMT

Also, how do I add 2 alternatives to the choice design, the kind you shared? Thanks again.

Re: Two alternatives in choice experiment

Victor_G — Fri, 11 Oct 2024 12:14:57 GMT

Hi @aus22,

Ok thanks, the context is now clearer and you may need effectively a Choice Design.

In order to set up your design, you have several parameters to define in the Design Generation panel :

The "Number of attributes that can change within a choice set" is the number of factors that you can vary the levels between experiments (called "profiles") in the choice set. In my design, only 2 of the 3 factors vary between the 2 experiments in the same choice set (example for survey 1, choice set 1, X1 and X2 vary, but not X3 (fixed at L1)). This should answer your last question ?
The "Number of profiles per choice set" is the number of experiments in the same choice set that the respondent will have to choose from. In my example, I let this value to 2, but you can augment it if you would like to reduce the number of choice sets/surveys for example.
The "Number of choice sets per survey" : collection of profiles.
The "Number of surveys" : collection of choice sets.
The "Number of respondents per survey" : if you have several respondents, change the value to the number of people answering each survey, so that each respondent has its own column for the values. It will be automatically taken into account in the script "Choice" for the analysis, filling the "Subject ID" role with the "Respondent" value.

More infos on terminology here : Choice Design Terminology

Hope this response clarify the design and terminology ?

Re: Two alternatives in choice experiment

aus22 — Mon, 14 Oct 2024 02:29:00 GMT

Hi Victor thanks a lot this is most helpful. A follow up question please. How can I select the attribute that does not change? Thank you.

Re: Two alternatives in choice experiment

Victor_G — Mon, 14 Oct 2024 19:57:26 GMT

Hi @aus22,

The attribute that does not change will be randomly selected for each choice set, so that every attribute has the same probability of staying fixed or changing for each choice set. I don't know a way to predefine the attribute that won't change in a specific choice set ?

I hope I did undertsand your question ?

Re: Two alternatives in choice experiment

aus22 — Mon, 14 Oct 2024 18:36:55 GMT

Got it thank you Victor most kind!!

Re: Two alternatives in choice experiment

aus22 — Tue, 15 Oct 2024 10:35:14 GMT

Hello Victor, I was able to generate a study design, see enclosed. Table 1 is what the software gave me. I replaced the choice set column with the product alternatives that I wish to offer in the choice sets - thats Table 2. Am I interpreting this correctly? Also, am I able to use this design to simulate data and estimate my utility functions? You will see the last two columns of Table 1 related to that trial. Thank you indeed.

Re: Two alternatives in choice experiment

Victor_G — Tue, 15 Oct 2024 12:59:54 GMT

Hi @aus22,

Please add your JMP file instead of Excel file, as all scripts are lost when using Excel files (unless converting files from JMP to Excel with the ).

Table 2 seems to be a transpose of Table 1, or did I miss something ? In any case yes, table 2 is a comparative file between products in column, separated in 3 choice sets with 4 products in each choice set.

Yes you can simulate data when creating your design, go into the red triangle next to "Choice Design" and click on "Simulate Responses" :

It will create formula columns for probability calculations and choice simulated, depending on the marginal utility values entered :

You can then launch the script "Choice" (or go to Analyze, Consumer Research, Choice) and analyze the simulated Choice response. You will have access to various Profilers (utility, probability profiler) and comparison analysis, as well as Willingness to Pay metric if you have specified your variable Price as Continuous numerical factor type.

You can practice with the dataset "Laptop Results.jmp", available in the Sample Index (Help, Sample Index) and follow the analysis method on this use case here : Example of Making Design Decisions

Hope this answer will help you,

Re: Two alternatives in choice experiment

aus22 — Tue, 15 Oct 2024 18:51:51 GMT

Hi Victor, Yes table 2 is a transpose. Does this study design make sense? Also, I did try to estimate the simulated data and it gave me all zeros. Any thoughts please? Thank you so much for following up on these two questions. Regards.

Re: Two alternatives in choice experiment

Victor_G — Tue, 15 Oct 2024 19:04:22 GMT

Hi @aus22,

Can you provide more context and infos ? JMP file and the steps you have done leading you to 0 ?

Using your Excel file and the Choice simulated column, I have a "normal" analysis window :

You may encounter "a lot of zeros" for the factors estimates if your response is random binary 0/1 and not influenced by specific factors. Also if you are trying to fit a complex model to a small and simple design, you can obtain zero on logworth, so no significant p-values :

In this case, the correlations between terms in the model are high, preventing the precise estimation of effects and inflating std error. It's becoming almost impossible to evaluate statistical significance in this situation, so p-values are equal to 1 for each term.

Attached please find your file.

Hope this answer may help you,

Re: Two alternatives in choice experiment

aus22 — Wed, 16 Oct 2024 02:26:50 GMT

Hi Victor, I understand, thank you. I am still unclear on whether the designs I have generated are correct or not. Can you please help with the enclosed file:

Choice Profiles - Part 1 Oct. 15 - This design's objective is to understand the factors that will influence respondent choice between two alternatives with three attributes.

Choice Profiles - Part 2 Oct 15. - This design's objective is to understand the factors that will influence respondent choice between 4 products with those similar factors except we need an additional one that specifies the type of materials used in the product. We need to offer 4 alternatives given the 4 different types of materials that can be used to create this product.

Can you please advise if these designs are ok? Thank you so much.

Re: Two alternatives in choice experiment

Victor_G — Fri, 18 Oct 2024 07:26:51 GMT

Hi @aus22,

It may be difficult to assess if the designs are acceptable for your needs, as I don't know your industry, context, topic, etc...

However, some questions that may help :

In your two designs, I don't see information about the respondents. Do you only have one ? If yes, how sure and confident are you that this person will be consistent, reliable, accurate and representative of your consumers (or studied population) to discriminate the choices ? Having several respondents may also help understanding the generalization of the outcomes and conclusions, regarding a particular benefit of one of the factor level in the choices.
Are the products between the two studies the same ? If yes, why not creating a design for part 1 with also the material type ? Including this factor early in the design could help discriminate the material type quickly, and understand its relative importance compared to other factors.
Think iteratively. I think using a first design as "screening" design might be helpful, but you can augment this design and use the prior information generated from the first design to confirm and precise the relative importances of factors, estimates, utility, etc... You can find an iterative example here : Design the Final Choice Experiment Using Prior Information
Also using an iterative approach instead of a "one-shot" design is helpful to manage expectations, as you can always augment and enrich your understanding and knowledge about the products.

All the best for your experimentations !

I hope these few remarks may help you to solve your problem,

Re: Two alternatives in choice experiment

aus22 — Fri, 18 Oct 2024 11:22:36 GMT

Hi Victor, thank you so much for your thoughtful advice. I actually did not include the no of respondents in the info I sent. We plan to collect data from at least 200 respondents. I like your suggestion of thinking through an iterative approach.

JMP question: One challenge I am facing is that I am unable to get the software to provide a design for only one survey with that number of parameters. So would it be ok for me to use Survey 1 and Survey 2 in my enclosed designs to be randomly assigned to each of my 200 or so respondents? Thank you as always.

Re: Two alternatives in choice experiment

Victor_G — Fri, 18 Oct 2024 15:08:44 GMT

Hi @aus22,

Related to your answer and JMP question :

If you have 200 respondents, I would consider providing several random surveys to random group of people, instead of one survey for all participants. Having different choices sets in different surveys enable to test different combinations in each survey, and help generalize the findings and better estimate relative importance of the factors.

It's a compromise to find : avoid having a specific survey for each participant (which could lower the precision of estimation of the factors relative importance), but avoid having one survey for all participants (which could compromise generalization of the results as less combinations are tested, and could create some noise due to inconsistent results from some participants).

As an example, you could maybe create 4-5 surveys for each topic, so that you collect results from groups of 40-50 participants ?

Re: Two alternatives in choice experiment

aus22 — Fri, 18 Oct 2024 17:45:42 GMT

Hi Victor, thank you for that suggestion, I appreciate it a lot. I have never done multiple surveys for the same respondent so not very confident of how to develop the survey instrument, and do the estimation. If you have any resources from JMP that you can share I will truly appreciate it because what you are suggesting does sounds like a better approach. Thank you!!

Re: Two alternatives in choice experiment

Victor_G — Fri, 18 Oct 2024 19:11:27 GMT

Hi @aus22,

To be sure there is no misunderstanding, the suggestion I made was to create 4-5 different surveys for 4-5 groups of different (random) respondants, not that each respondants answer to multiple surveys.

Here is a snapshot of what I meant in terms of parameters in the Choice design platform :

When creating this design, you'll have 200 independent respondants, distributed in 4 groups of 50 respondants, each group answering a specific survey. Please find attached the design used here as an example, to see the repartition of 200 respondants in 4 groups and the combinations and choices set used in each survey. The platform "Choice" for the analysis takes into account the survey number and respondent ID, so analysis is quite straightforward and simple, and already prepared with the "Choice" script present in the data table generated by the design creation platform.

I hope this response will help you,

Re: Two alternatives in choice experiment

aus22 — Mon, 21 Oct 2024 13:59:04 GMT

Hi Victor, thank you, thats what I thought earlier on so I am glad you clarified. I will give this a try during the week. Again thank you so much! Best.