I'm here with my colleague Jacob Rhyne,
to talk about some recent work on the new user interface in JMP Custom Design.
The primary focus of our work
was a usability study to evaluate the new interface.
First, I'm going to talk a little bit about the current interface
and our motivation for the change.
We think of experimentation as a workflow.
In the very first step, we define our factors, responses, and goals.
This critical step determines so much in what follows.
Adding factors is a very important step when you're designing an experiment.
We learned that customers have some difficulty
with common tasks around defining factors.
There's confusion about adding, deleting or renaming factors.
They find it difficult to change the factor type, the factor levels,
even reordering the factors in the list.
Figure 2 shows the original User Interface for comparison.
The goals for this prototype were to decrease confusion,
to reduce the number of operations in the time that it took to achieve the result.
Our usability study wanted to compare different controls
that might be used to achieve these goals.
Figure 3 shows the prototype for the new user interface.
This prototype is dynamically configurable.
That is, we can, that will turn different user controls on and off.
This led to a prototype with 384 possible configurations.
It was impossible to include all of them in a usability study.
A practical study required a smaller, optimal design.
Looking at figure 3, especially along the top,
you see the User Controls
that we're going to primarily focus on in our usability study.
You see the Trash icon, the Delete icon,
the Plus and Minus buttons, the Undo and Redo buttons,
the Add N Factors control, and what we call our Buffet menu button.
In figure 4, you see just two of the 384 possible configurations.
In order to make sure that the usability study
was efficient and effective, we used Custom Design
to make most of the runs for our study.
In this case,
some of the User Interface controls were treated as a binary factor.
That is they were turned on or off in the prototype.
Other User Interface controls could have more than just two states.
Each time the subject was asked to perform several different tasks
repeatedly with different configurations for comparison.
In the table in the lower left,
you see all of the factors that were included in our study.
The first six, seven, have to do with controls
in the User i nterface that were turned on or off or to a different state.
The last factor is the one that determined
what we wanted them to do with a particular configuration.
That is we asked them to delete a factor,
change the type of the factor, or change its position in the list.
On the right in figure 5,
you see the completed Custom Design platform
with the response defined.
We're going to measure the time it takes for them to complete each task.
We're trying to minimize that.
All of the factor definitions, and finally, the model
that we want to use to quantitate the time to complete each task.
I'm now going to turn it over to Jacob.
Who's going to talk about the resulting Custom Design,
how that fit into our study, and our findings.
Thank you, Mark.
Mark has laid out how we use the Custom D esign platform in JMP
to set up our usability study.
In figure 6, you can see, all the runs in the study, for the first participant.
I'll note that we did include a participant and Run Order in our design
as covariates because we wanted to include these in our analysis as well.
You'll also note from figure 6 that we included a few,
what we call, special tasks in our study.
These were tests we knew would take longer to complete,
and we didn't want to ask participants to do these more once.
If you look at the final task of the of the study,
these were, special exercises that we ask the participants to complete
a series of tasks using a provision of the prototype
and the same series of tasks using the Custom Design platform.
This allows us to get a direct comparison
between the new configurable prototype and Custom Design.
To apply this Custom Design, we developed a custom JSL application
that allowed us conduct an unmoderated usability test.
Now I'll give an example of how this works in figure 7.
From the top of figure 7,
you can see that, for Run Order 2, the task was to delete.
As you can see in the second part of the screenshot,
the instructions the participant saw in the study, instructed them to click
the Launch Prototype button and then delete the factor charge.
When the participant clicked the Launch P rototype button,
the version of the prototype that appeared here in the third part of the screenshot,
that's entirely determined by the other factors in our study.
For example, you could see the Trash icon is set to off.
That means you don't see the Trash can icon,
down here in the prototype.
The Delete icon is on, so you do see the Delete icon.
For every exercise, in our study,
the version of the prototype that appears and the instructions
that the participant would see
is entirely determined by our Custom Design.
Let's look at some of the qualitative results from our study.
In our usability, study scripted we included an integrated questionnaire
at the end where we ask participants to get us some feedback
on what they liked, what they didn't like, et cetera.
I'll highlight, a few of the options in figure 8 here.
We ask participants to give us feedback on whether they liked using
the Plus button or the Add N Factors control
to add factors, and you can see from the top of figure 8
that participants overwhelmingly preferred using the Plus button
to the Add N factors control.
We also ask participants how they liked, Deleting factors,
either using the Delete icon or the Trash icon.
In this case, the Delete icon was overwhelmingly preferred.
We also asked participants, what was their favorite Row icon
that you could see beside the rows in the factor list.
The Pin icon was the most popular with, giving five of the 10 votes,
compared to the Oval icon, which only got two of the 10 votes.
Then, finally, I point out if you look down towards the bottom of figure 8,
you can see that the participants overwhelmingly voted
that the User I nterface of the new prototype was easy to learn.
It was the new interface was also greatly preferred,
to the original factor listing Custom Design.
Now let's look at some of the quantitative results from our study.
We got these results by using the generalized regression platform
available in JMP to fit models
for the time it takes to complete the task, and the factors in our model
or the effects on model, excuse me,
or the different elements of the prototype.
If you'll look at figure 9, for the output of our generalized regression,
you can see, that the factors, Row States, Trash icon,
and Pin C olumn Type were found to have a significant impact on time.
You can also see, from the profile or the Run Order,
in this case, did not have a significant impact on time.
Let me draw your attention, to the Pi n Column Type F actor.
You can see that the time to complete, the task of changing a factor order
was minimized when the Oval icon was used to complete the task.
I'm pointing out this Pin Column Type Factor specifically
because in this study, the way participants would change
the order of a factor is they would click the Row icon beside the factor
in the factory list and drag that icon to the appropriate spot.
In this case, the Pin Column T ype is what I wanted to focus on.
Looking at figure 10, you can see that,
participants, when they were asked to tell us their favorite,
only two of the 10 participants reported liking the Oval icon.
When it came to completing the task, that participants interacted
with the icons the most, the Oval icon was actually the best- performing icon.
Now let's look at the results, for the task of deleting a factor.
In this case, participants would be asked to, given version of the prototype,
remove one factor from the factor list, and they could do that by either using
the Delete icon, the Trash icon, or they could have the option to use both.
Again, we fit models using a generalized regression
with time as the response, and you can see that,
the icon was used and Run OrderThe bot had a significant impact on time.
The time, for this task was minimized when the Delete icon was used as opposed
to using the Trash icon or have any option to use both.
In contrast, to the previous slide,
our quantitative results here match the qualitative results
because as you can see in figure 12,
the participants overwhelmingly preferred the Delete icon, to the Trash can icon.
I'll end this study by commenting on the results of the last two exercises.
The last 2 exercises in the study,
we had we had participants complete a series of exercises using the prototype
and then complete the same series of exercises using Custom Design.
The instructions for the exercises were the same.
The only thing that was different is in one case you use the prototype,
and in the next case, you use Custom Design.
When participants were giving the prototype,
it took them an average of 68 seconds to complete the task.
When participants use Custom Design platform,
it took them an average of 316 seconds to complete this exercise.
I'll also note this 316 seconds is right- centered
because a couple of the early participants
that we gave the study to reported that they gave up.
A fter this, we started popping up a notification after five minutes saying,
"We thank you for completing this exercise.
You can move on to the next one."
That's all we have to share today.
Want to thank you for your interest in our poster,
and please let us know if you have any questions.
