From Development to Testing: A Journey of Creating and Validating a JMP Add-In

Hello everyone. Thank you for joining me today. Today I'll be giving a talk on this project I worked on, and the title of this talk is From Development to Testing: A Journey of Creating and Validating a JMP Add-In.

Just a little bit about myself. I'll first introduce myself. I am a graduate student currently pursuing a PhD in the Mathematical Sciences at Clemson University in the US. My research interests are broadly within Experimental Design and Preference Learning. I've been interning with JMP's DOE & Reliability team since the summer of 2024. During my time there, I've worked on projects related to both discrete choice experimentation and software testing. Two pretty different fields. That's enough about me.

I'll go ahead and give an overview of today's talk. First off, for the first section of this talk, I'm going to be talking about the motivation for creating this add-in that we built over the summer. The motivation for this is we were conducting some usability studies for JMP's program Easy DOE, and we're trying to learn user interface preferences of individuals within different demographic groups of the user base. After that, after talking about the motivation for creating this add-in, I actually talk about the add-in and give a demonstration of it. Then lastly I'll talk a little bit about testing the add-in through the JSL unit testing framework to validate the application or the add-in that we created.

Here's the first section. The motivation for creating our add-in. As some of you may know, there is a program within JMP called Easy DOE. This program is used for helping people, for guiding people through the entire process of designing and analyzing an experiment. This Easy DOE workflow has a couple different components. The experimenter starts off with defining the factors and the responses that they're interested in. After that they select the model that they want to use to analyze the data that they're going to collect. After selecting a model, they design an experiment for their model.

Once they design that experiment, they can go and run the experiment and then input the data they collected into the Data Entry tab. Once they have their data collected, they can analyze the model. Analyze the data and fit the model. Lastly, they can use that model to do predictions and use it to select optimal factor settings, if that's what they're interested in. Then lastly they can view a summary report of the model.

One particular thing we'll be interested in looking at today is this Analyze tab. As some of you that have used Easy DOE know, the Analyze tab looks like this, as presented in Figure 2. What we were interested in over the summer was seeing what changes could be made to certain parts of the Analyze tab to make it more usable for experimenters and users that used Easy DOE. We ultimately proposed a couple different versions of the UI that users may be able to see, and we wanted to learn which one of these versions was the most preferred.

This UI design, how we had it designed is, it had a factorial structure. The particular part of the UI we were interested in was this hover help. This hover help has a couple of different messages that informs the user about some statistical properties of their model. Some of these messages in the hover help settings include the Entered Message, whether a term was entered in the model. We had two different versions for that. The Significance Message, whether the term was significant or not. A Heredity Message, why was a certain term included in the model, even though it's insignificant?

Oftentimes, there's this property called heredity, where if an interaction term is significant, but the main effect isn't, we'll include the main effect anyways. Lastly, one last factor of this hover help setting was the location of the hover help. Is it contained over a Confidence Interval or is it over the Preview Column?

We were wanting to learn, which of these settings did users prefer the most? As I was saying, you saw that we had four factors. Two of those factors or three of those factors had two levels, and then one of the factors had three levels, so there were a possibility of diff of 24 different versions of the user interface for the hover help settings.

Another thing is, within Easy DOE, oftentimes users of Easy Doe are people that are often either new or only have maybe some moderate experience in design of experiments. But sometimes we also have people who are experienced in design of experiments that use Easy Doe, and they may either use it for personal use or for teaching others. We have this different demographic groups that use Easy Doe. We wanted to learn, which hover help settings do different demographic groups prefer, so that we can tailor the design of the hover help to each demographic group.

How we wanted to approach this problem was through the framework of contextual discrete choice experimentation. I'll talk a little bit about what this framework is. First off, I'll talk about the data collection process. In contextual discrete choice experimentation, you have a group of participants, and you have some covariate information of these participants. Or for example, we call it covariate information or contextual information. Oftentimes it could be demographic information that you have on your users.

What we do in this framework is we subject these different participants to a questionnaire, where in each question they are asked to compare two or more alternative designs and pick the one that they prefer more. The data collection process is pretty intuitive, and it reflects how people may choose options in real life.

Now I'll talk a little bit about modeling and analysis for contextual discrete choice experimentation. Once you collect the data, you can then fit a utility model to this data that involves, not only the factors of the thing that you're studying, which in our case are the hover help settings, but also contextual information coming from the participants. These two, when combined together, can give you an idea of what the preferences of different hover help settings are for different users, coming from different demographic groups. Ultimately, once you fit this utility model, you can then use it to learn or find out which of the designs of the hover help are most preferred for each demographic group.

One important component of contextual discrete choice experimentation is the design of these experiments. How do we pick the questions to present to the participants? How we do this, if you're familiar with regular discrete choice experimentation, oftentimes to get a good design for discrete choice experimentation, you oftentimes need some prior information on the user's preferences.

How we do that in contextual discrete choice experimentation is we first conduct a so-called screening study, where users, we first start out with some number of participants, and we have their contextual information or demographic information, and then we subject them to a questionnaire and gather their preference data. From that screening study we fit a model for the screening study that characterizes the user's preferences from the screening study. We can then use this model to understand preferences across different demographic groups.

Thus, we can use this model to create Bayesian priors that can be used to create Bayesian D-Optimal Designs for the questionnaire. We can use that to create questionnaires which are targeted to each demographic group in such a way that we can gain more statistical information in a more efficient manner from that group. That is the methodology that we wanted to… how we wanted to approach this problem. We wanted to create very statistically efficient questionnaires to give to our participants, so that we can more efficiently gather information from them and learn preferences across different demographic groups.

I'll talk here for a bit about actually creating this add-in. First off, for those of you that don't know, JSL is JMP Scripting Language. It allows users to extend the functionality of JMP by allowing users to do a whole lot of things such as running routine multistep analysis analyzes or fitting a specified model to data, which is generated on a daily basis. JSL really allows users to extend the functionality of JMP. What a JMP Add-In is, is it's a JSL script which extends the functionality of JMP which other JMP users can download and use.

Why did we create a JMP add-in for our particular problem? First off, we were conducting multiple of these usability studies. We were conducting the same usability study, the one for the hover help setting, at different points in time. We were gathering more and more data over time, and we would use that to refine the experimental designs or the questionnaires that we sent out to people. We're also conducting different usability studies as well, for example, where different factors of the user interface could change. For example, one of the usability studies was on the color scheme and location of notification messages on the Analyze tab.

Another reason for creating this add-in was that creating these contextual discrete choice experiments could be tedious as it were often required calculating those prior means and covariance matrices to create those Bayesian D-Optimal Designs for each group. We found that this could be automated through JSL scripting.

Then lastly, this framework of contextual discrete choice experimentation, you can use it for a very wide range of choice tasks. It's not confined to just usability studies, but for any kind of study where you need users to state their preferences among multiple options. We wanted to create this add-in not as only a way to help ourselves, but also so that other people could use the add-in as well to solve their problem.

I'll talk a little bit about the workflow of our contextual discrete choice add-in. First off, what the user or experimenter needs to do is they need to collect data from a screening study, where the screening study should include information such as what questions did you offer to the participants? What were the attributes? The profile attributes. The factors of the thing you were studying, as well as subject covariates.

Once you have this data, then you fit a screening model to the data. In the screening model the experimenter needs to specify both the profile effects. In our setting, in our case, that was the hover help settings as well as subject effects. This allows the experimenter to create a utility model. Then this utility model is estimated using JMP's Choice Analysis Platform.

Once that is done, the user then selects a demographic group for further study. They do that by selecting levels of each of those subjects effects or contextual factors. A combination of those levels specifies a group for further study. Once they select that group, then a Bayesian D-Optimal design or a Bayesian D-Optimal questionnaire is constructed for that group, so that the experimenter can further study that group in a statistically efficient manner.

Now I'll go ahead and give a demonstration of the contextual discrete choice experiment add-in. I'll go over here and open up JMP and go to my Add-Ins. I'll select the data table. This data table contains information… I'll show what it looks like real quick. This data table contains information on previous usability studies that were conducted for the hover help. You can see we have information such as, which option did the user choose? What were the options presented? We have some contextual information on the user, such as their experience level in design of experiments, as well as their reason for using Easy DOE. Is it for personal use or for teaching others?

Within this… I'll go here and open this up. We'll first put in the response indicator which is the choice the user made, and we'll also set the choice set ID and the subject IDs. Then here we can go ahead and input the profile effects, which in our setting were those three messages in the location of the hover help. If we wanted to, we could add interaction effects, if we wanted to. But for right now I'll just keep it as a main effects model. Also, we can add in subject effects such as the experience level and whether the users teach others or are using Easy DOE for personal use.

You can see that a lot of the information here looks just like the setup for the Choice Analysis Platform. When we hit Run, what happens in the background is the Choice Analysis Platform is pulled up, and it analyzes the screening model for the screening study. Of course, we can interact with this just like we would interact with the Choice Analysis Platform. For example, we can pull up a utility profiler, and for example, we could see that experienced users, that are teaching others, really prefer a certain option for the heredity message, which is this second alternative here.

Once we look at that, we can pull up this component here which asks us to select a demographic group for further study. Say I was conducting these studies and I wanted to study the expert users of Easy DOE, so those who are experienced in design of experiments, and I'm interested in when they're teaching others. I want to make a questionnaire to learn the preferences of expert users who teach others. I click Make Design and that pulls up the Choice Design platform in such a way that information that we already entered in the Choice Analysis stage or the Screening Study stage is brought automatically into the Choice Design platform.

We have our attributes, we have our model here for this particular group which was a main effects model that we specified. The most important thing here is that using the model from the screening study, that screening model, we were able to calculate Bayesian priors for the prior mean and the prior variance for the particular group that we were interested in studying.

Once we created this prior, we can then create a so called Bayesian D-Optimal Design by clicking Make Design, and then we get a design here that is now tailored for the specific group we're studying. The construction of this design took into fact the prior information that we had on this group's preferences. Now this design is tailored to the group that we want to study further. That's my demonstration of the contextual discrete choice experimentation add-in. I'll close out some of these, and I'll go back to our slides.

Continuing on, the last thing I'll talk about now is testing the add-in through the JSL unit testing framework. What is unit testing, first, for those of you who don't know? Unit testing involves testing individual components of a software application to identify errors. In particular, automated unit testing allows one to frequently run unit tests on a software application to identify and correct newly introduced errors through the development process.

In particular, JMP has a unit testing framework called the JSL unit testing framework which is tailored for JSL applications. The most important function within this framework is the ut assert function, which allows users to test certain components of JSL application. The user has what they believe is the correct output, which should be the correct output, and that's what is called the expected value. They test that against the real output of their application. If they match, that means that the test case passed, and if it doesn't match, that means the test case failed. When the user writes a script with these ut assert functions, and they run it, their report will be returned letting the user know if all the test case passed or which ones failed, so that these can be further investigated.

We tested quite a few things within the contextual discrete choice experimentation add-in. For example, one of the things we tested was our cross button for adding in interaction effects. Was that working correctly? We had an idea that if we pick two certain options and hit the cross button, we know what that should look like, and so we can test that against the actual output to make sure that the system is working correctly.

Another important thing we tested was, when you're selecting a demographic group for further study, we want to make sure that a message would pop up if the experimenter left one of the options blank. We did test and made sure that this error message was popping up as required.

Lastly, this is an important one. We wanted to make sure that the prior information that we were loading into the choice designer was correct. We tested that. We tested that against the output of the Choice Analysis platform. If those two matched, then we knew that we were bringing this prior information into the Choice Designer correctly, and if there was a mismatch, we knew that something was going wrong. Thankfully, we found that we were correctly loading in the correct prior information.

Now, one of the most important things for this automated unit testing is when you go to refactor your code. Automated unit testing allows one to refactor one's code and check if any errors were introduced. By refactoring code, I mean making changes to the code through the development process. If you make a change to your code, how do you know whether or not you broke something else that's not related to what you were changing?

A certain example we found was that one of the test users of this add-in, they found that when one of the contextual factors had a value label, then what would happen was that the list for that contextual factor, it wouldn't display correctly. It wouldn't display at all, really.

We went back and fixed that error. But we wanted to make sure that by fixing that error in the way we did, we didn't break something else. We could do that. We checked that by rerunning our unit testing script. Thankfully, after running our unit testing script, we saw that no new errors were introduced and so that helped ensure that we didn't break anything else. Our proposed fix for that value label issue did correctly fix the issue, as you can see on the right side of this picture here.

This unit testing is important because, one, it can let you know whether there's errors in your code. Two, it allows you to refactor your code and make sure you didn't introduce any other errors. Three, as JMP updates, certain internal things within JMP are going to change, certain platforms might change. If you have this unit testing script, you can always test your add-in when JMP is updated to make sure that something that changed within JMP didn't or did affect your add-in.

As a conclusion of today's talk, I introduced the problem of designing contextual discrete choice experiments to help learn user Interface preferences for Easy DOE across different demographic groups of our user base. Then I demonstrated how JSL scripting can allow us to create a very convenient to use JMP add-in for constructing these discrete choice experiments. Lastly, I discussed the importance of… once you either during or after the process of making your add-in, you'll want to perform some unit testing as a way to help detect errors during and after the development process.

That concludes my talk for today. Are there any questions?