Testing is a critical component of complex systems engineering, yet test engineers are rarely able to evaluate every possible combination of system inputs. Growing system complexity, limited budgets, and tight schedules demand more strategic approaches to test design. The challenge becomes even greater when certain inputs or factors are expensive or time‑consuming to change, forcing engineers to carefully balance test coverage with practical constraints.

In this talk, we explore how covering arrays can be used to efficiently test complex systems while maintaining meaningful coverage of the input space. Covering arrays are a powerful combinatorial testing tool that enable engineers to achieve specified interaction coverage using far fewer test cases than exhaustive testing. We then extend this approach to address the presence of hard‑to‑change factors by introducing covering arrays with split‑plot structure, allowing test plans to reflect real‑world testing constraints. Using JMP, we demonstrate how these structured covering arrays can be constructed in practice.

Finally, we present a real application of this methodology for testing a JMP add‑in, showing how incorporating split‑plot covering arrays can significantly reduce testing time and effort while preserving confidence in system behavior.

 

 

Hello, everyone. Thank you for joining me today. My name is William Fisher, and I'm a developer on the DOE and Reliability team at JMP. Today I'll be giving a talk on this poster, which is Testing Complex Systems with Hard-to-Change Factors Using Covering Arrays. In this talk today, what I'm going to be going over is a testing problem that I had worked on during my internship at JMP. This testing problem involves testing a JMP add-in, which is the JMP XGBoost add-in. In this talk today, I'm going to present some of the challenges that I came across while testing this add-in. In this context, by testing, what I mean is trying to identify faults in this add-in and validating that it functions as we would expect it to. What this talk is going to be about today is going over some of the challenges I came across while testing the JMP XGBoost add-in, which is an add-in that has been available for a long time on the JMP Marketplace, and I'm also going to be talking about how I addressed some of these challenges that I came across. I'll get started and go into the actual example.

One of the things I worked on during my internship was testing and validating certain functionality of the JMP XGBoost add-in, which had been developed quite some time ago. What we were interested in validating and testing was the auto-tuning functionality of this add-in, as well as the auto-tuning features interaction with the built-in prediction profiler. If you've ever downloaded and used the JMP XGBoost add-in, when you open it up, you'll see that one of the first things you'll come across is this auto-tuning user interface, which basically allows users to select some settings to perform auto-tuning on a couple of hyperparameters for the underlying machine learning model in XGBoost. Once auto-tuning is complete, the users are able to get access to a prediction profiler, which allows them to see the relationship between the predictor variables in their dataset, as well as the response that they're modeling. One important aspect of developing an add-in is actually testing it, thoroughly testing it to make sure that it is functioning as expected. What we wanted to do here was, with this auto-tuning functionality enabled, we wanted to test that the prediction profiler was giving the correct numerical output. But this testing… Because we want to make sure that the software is functioning as it should be, so that when users use it, they can have more trust in it.

One of the testing challenges that came up when we were testing this add-in was that there's a large input space to this system. You imagine here in the auto-tuning feature, there's 10 settings. The settings, 8 of them take on this interval-type data where you specify an interval to search for the best hyperparameter settings. On top of that, given a dataset, the prediction profiler can also have numerous predictor variables. The input space for this testing problem is pretty large because there's a lot of different values that users can put into this system. On top of that, a very unique challenge for this testing problem is that the testing time can take quite a while. Depending on what the users specify for their auto-tuning settings, and depending on the dataset they have, auto-tuning could take a long time. It could take 10 minutes or maybe 15 minutes, depending on the settings. If you have any experience with design of experiments, you can kind of think of these auto-tuning settings, all these settings associated with the auto-tuning procedure, as hard-to-change factors or inputs in this testing problem. Whereas the factors and inputs for the prediction profiler are a lot more easy to change. They don't take much time at all because the calculation is very quick.

Lastly, one last challenge that we had was actually determining the oracle. Whenever you're testing a software system or any type of system, you need to know what the expected result or expected output of the system is as a baseline to test what has been implemented in practice. How do we actually… You've seen the system here, we had the total of 27 different factors or inputs, and we classified… The hard-to-change factors were all associated with the auto-tuning procedure, and the easy-to-change factors were all associated with the prediction profiler. Now, one thing I'll say is to actually approach this testing problem, what we did is we decided to perform testing using the dataset that comes with the XGBoost add-in, which is this powder metallurgy dataset. That has 17 predictor variables, so those will be associated with the prediction profiler.

To handle the large input space, what we did was we preselected certain values of each of the inputs to test, and we decided to use covering arrays, which are available in JMP, to help us select which input combinations to actually test. That part can be directly solved using JMP. But the challenge that gave us… The one challenging thing was the long testing time. Directly using covering arrays from JMP, it may be the case that we would… The covering arrays give us a set of test cases to test, but it may be the case that the settings for the auto-tuning procedure are going to change across every single test case. That could take a lot of time for testing, resetting those auto-tuning parameters every single time. What we did is we took a little inspiration from design of experiments, and we decided to create covering arrays that have a split-plot structure. If you know from design of experiments, when you have hard-to-change factors, you often use split-plot designs. For this testing case, we decided to use covering arrays with a split-plot structure to minimize the number of times that we have to reset the auto-tuning settings. Lastly, for the oracle, we decided to use the prediction formula, which can be added as a column in the dataset. We use that prediction formula as a pseudo-oracle so that we can test the output of the prediction profiler against the output of the prediction formula just to make sure that those two values are close together for whatever settings of the prediction profiler we test.

In my previous slide, I mentioned for testing, we wanted to use covering arrays. I'll go a little bit more in depth on what covering arrays are. What we did for our testing problem is we identified combinatorial testing as a suitable framework for approaching our testing problem. Combinatorial testing is this testing method often used to find faults in a system under test by checking if any t-way combinations of system inputs or factors induces a failure for some chosen value of t. This is typically achieved by selecting a suite of test cases that cover all possible t-way combinations of the input settings. Just to visualize what I'm saying here, here in the second image to the right, I have a suite of tests. There's a simple testing problem with three factors, each having two levels. For each test case, what you would do is take the test case, evaluate it under the system under test, get the output, and compare it to your oracle value and see if it's a pass or a fail. Given this data from a test suite and the pass-failure data for each test case, we can investigate which t-way combinations are potentially causing failures in our system so that we can investigate those further and correct them.

This combinatorial testing technique, it often relies heavily on the use of a particular type of design called covering arrays. What covering arrays are, they're a certain type of array, they're available in JMP, and they are an array with N rows and K columns. They satisfy this property that for any selection of t columns… If you specify parameter t, for any selection of t columns, all t-way combinations of those symbols from those columns appear at least once. This is often applied… Covering arrays are often applied in combinatorial testing because they give you a way to cover all t-way combinations of the input settings so that you're really covering the entire test space in a strategic manner. We often want to find covering arrays with a small number of test cases because testing can be a very time-consuming procedure. I have an example of a covering array here, and the bottom image in the center here, this is a strength-two covering array on five factors. The first factor, X1, has three levels, and the remaining just have two levels each. This is a strength-two covering array because for any two columns I pick, all possible two-way combinations appear at least once.

To actually create a covering array in JMP, it's pretty simple. You can just go to DOE, Special Purpose, and go to the Covering Array platform. The user interface looks a lot similar to other DOE platforms, but you can specify the strength t in the Covering Array platform. Once you created a covering array and gather some data on the failure or the pass or fail data for each test case, you can easily analyze it using this Analyze button, and it'll tell you which t-way combinations are likely to be causing a failure in your system. Those are covering arrays and combinatorial testing. These are what are going to be what we're going to use heavily for actually trying to solve our testing problem for the XGBoost add-in. We're going to use these covering arrays to guide us on what we should test. But if you remember, there was a very particular challenge that we faced when testing XGBoost, in particular testing the auto-tuning procedure. If you remember, in the XGBoost testing problem we have, you first do auto-tuning, which can be very time-consuming. Then once you complete the auto-tuning, you get access to the prediction profiler, and you can test different values of the predictor variables and check the predicted response against our pseudo-oracle.

Now, using just traditional covering arrays, it might be the case that we would have to reset the auto-tuning settings numerous times, which could be very time-consuming. What we did is we came up with a way to create covering arrays with split-plot structure so that we could reduce the time spent on testing. We basically wanted to try to minimize the number of times we are changing the auto-tuning procedure settings while preserving the coverage of the test suite. How we do that… How we actually do that is we first identify which system inputs or factors are hard to change and easy to change. Like before, we knew that everything associated with auto-tuning was hard to change, and everything associated with the prediction profiler was easy to change. What we did next is we created a covering array for the hard-to-change factors. That gave us a… We created a strength-two covering array for the hard-to-change factors, which were the auto-tuning settings in the XGBoost testing problem. Then what we did next is we created a covering array for the easy-to-change factors. The factors associated with the prediction profiler, as well as this pseudo factor that we introduced.

The pseudo factor has the same number of levels as the number of runs in the covering array from step two. This pseudo factor will be used for constructing whole plots, which I'll describe what that means in step four here. In step four, once we have created this covering array on the easy-to-change factors and the pseudo factor, what we'll do is we'll create a new data table in JMP and copy over the response and pseudo factor columns from step three. We'll also add columns for all the hard-to-change factors next to the pseudo factor column, and then we'll copy over the easy-to-change factor columns and all their values from step three. What we do next is, depending on the level of the pseudo factor… I did forget to mention in step three, we sort the data table with respect to the pseudo factor to form these nice blocks or whole plots. But going back to step four, what we do is we look at the pseudo factor column, and what we'll do is if we see… Each level of the pseudo factor column is associated with some run in the covering array created in step two. If we see a level one for the pseudo factor in a certain row, we will place the run from the covering array created on the hard-to-change factors in the columns associated with the hard-to-change factors in this larger covering array.

You can see that here, after we create this… After we do this expansion of the pseudo factors, you can see that this array here has a split-plot structure. You can see that the hard-to-change factors, from max depth to number of inner folds, they are held constant across a certain number of runs. That's our whole plots there. Within those whole plots, the easy-to-change factors are allowed to vary freely. How we would use this test suite in practice is we would go to our XGBoost add-in. For the first whole plot, we would put in these settings marked in the big red square here and let the auto-tuning procedure proceed. Then once we're done with auto-tuning, we'd go to the prediction profiler and then test these five test cases, starting from formation method down to the last predictor, P5. This is what we did in JMP 19. But coming in JMP 20, we actually built this procedure into the Covering Array platform in that users can now specify which inputs or factors are hard to change, and we automatically create a covering array that has that split-plot structure while satisfying the user's coverage requirements.

Closing off, going back to the actual testing problem, what we wanted to do is we wanted to show that for testing problems that have these hard-to-change factors, you can really benefit from taking advantage of that structure. You can really take advantage of the presence of hard-to-change factors if you create a test suite that takes that into account. To actually showcase that, what we did for our XGBoost testing problem is we created two covering arrays. One covering array was a traditional covering array that does not take into account the presence of hard-to-change factors. That covering array, so it had 27… There's 27 factors. That covering array was able to… We were able to create a covering array with only 32 test cases using the traditional method. But because that covering array didn't take into account the presence of hard-to-change factors, those hard-to-change factors were reset 32 times. Whereas in our method, we create a covering array with split-plot structure. That covering array had 75 test cases. But because of the fact that we took advantage of the hard-to-change factors, we only have to reset the hard-to-change factors, those auto-tuning settings, only 15 times.

What does this mean? First, using our method, we saw that we actually had higher coverage of the testing space. By construction, both of those covering arrays were strength-two. They were covering all two-way interactions between the input settings. Our covering array with a split-plot structure, it had higher three-way coverage as well as four-way coverage compared to the more traditional covering array that doesn't take into account the hard-to-change factors. A very important advantage we saw was in the testing time. The traditional covering array, the total time it took to use that to go through all the test cases that it created, it took 104 minutes, and that's because we had to reset the auto-tuning settings 32 times. But our covering array with the split-plot structure, we were able to complete testing in just 61 minutes, and that was because we only reset the hard-to-change factors 15 times. In the background, the way we were doing testing, we were doing automated testing. There wasn't much of a… I wasn't manually going through each test case and testing it on my own. I was using JSL scripting to do a lot of the testing procedure. Now, even after adjusting for outliers in the auto-tuning times between these two test suites, the traditional covering array took 77 minutes to complete, while ours took 45 minutes.

We saw both an advantage. By taking into account that there's hard-to-change factors and creating a covering array that had split-plot structure, we were able to save quite a bit of time in testing. On top of that, it also aided in fault localization. What we were doing when we were testing was we would get the value from the prediction profiler and compare that against the value of the prediction formula in the data table. We were comparing them in terms of the number of significant digits where they matched. Of course for the first… They matched up to five significant digits across all the test cases. One interesting thing was that when you compare up to seven significant digits, we saw five failures coming from the test suite associated with the covering array with a split-plot structure. Something interesting there was that those five failures, each one was in a separate whole plot. What that told us was that it looks like there wasn't any failures being induced due to two-way interactions from the hard-to-change factors. That gave us a little bit more confidence in the auto-tuning abilities… In the implementation of the auto-tuning procedure in XGBoost. These other settings, checking the two numbers up to 8, 9, and 10 significant digits, we did see a lot of failures across that case. But I think that's pretty reasonable in that at some point, you're not going to have all of the digits between the pseudo-oracle and oracle matching exactly.

That is my talk here today. In conclusion, what we found was that in some testing problems, certain inputs or factors may be hard to change or reset in every test case. In that case, we proposed a way to construct covering arrays that account for these hard-to-change factors. These covering arrays have a split-plot structure, which is well known in design of experiments. We provided a way to create these covering arrays. We highlighted a process to create these covering arrays manually in previous versions of JMP. We also highlighted that for JMP 20, there's going to be an easier way to create these covering arrays, more user-friendly way to create these covering arrays with split-plot structure. Lastly, what we found was that accounting for hard-to-change factors in the test suite or the test plan can save a lot of time during testing by decreasing the amount of times the hard-to-change factors need to be reset. If you have a testing problem with hard-to-change factors and you have the right structure, you should really take advantage of that when you're creating your test cases in order to save time. Thank you for coming to my talk today.
0 Comments
Presented At Discovery Summit 2026

Presenter

Skill level

Intermediate
  • Beginner
  • Intermediate
  • Advanced

Files

Need help?
Published on ‎07-15-2026 03:39 PM by Community Manager Community Manager | Updated on ‎10-30-2026 03:35 PM

Testing is a critical component of complex systems engineering, yet test engineers are rarely able to evaluate every possible combination of system inputs. Growing system complexity, limited budgets, and tight schedules demand more strategic approaches to test design. The challenge becomes even greater when certain inputs or factors are expensive or time‑consuming to change, forcing engineers to carefully balance test coverage with practical constraints.

In this talk, we explore how covering arrays can be used to efficiently test complex systems while maintaining meaningful coverage of the input space. Covering arrays are a powerful combinatorial testing tool that enable engineers to achieve specified interaction coverage using far fewer test cases than exhaustive testing. We then extend this approach to address the presence of hard‑to‑change factors by introducing covering arrays with split‑plot structure, allowing test plans to reflect real‑world testing constraints. Using JMP, we demonstrate how these structured covering arrays can be constructed in practice.

Finally, we present a real application of this methodology for testing a JMP add‑in, showing how incorporating split‑plot covering arrays can significantly reduce testing time and effort while preserving confidence in system behavior.

 

 

Hello, everyone. Thank you for joining me today. My name is William Fisher, and I'm a developer on the DOE and Reliability team at JMP. Today I'll be giving a talk on this poster, which is Testing Complex Systems with Hard-to-Change Factors Using Covering Arrays. In this talk today, what I'm going to be going over is a testing problem that I had worked on during my internship at JMP. This testing problem involves testing a JMP add-in, which is the JMP XGBoost add-in. In this talk today, I'm going to present some of the challenges that I came across while testing this add-in. In this context, by testing, what I mean is trying to identify faults in this add-in and validating that it functions as we would expect it to. What this talk is going to be about today is going over some of the challenges I came across while testing the JMP XGBoost add-in, which is an add-in that has been available for a long time on the JMP Marketplace, and I'm also going to be talking about how I addressed some of these challenges that I came across. I'll get started and go into the actual example.

One of the things I worked on during my internship was testing and validating certain functionality of the JMP XGBoost add-in, which had been developed quite some time ago. What we were interested in validating and testing was the auto-tuning functionality of this add-in, as well as the auto-tuning features interaction with the built-in prediction profiler. If you've ever downloaded and used the JMP XGBoost add-in, when you open it up, you'll see that one of the first things you'll come across is this auto-tuning user interface, which basically allows users to select some settings to perform auto-tuning on a couple of hyperparameters for the underlying machine learning model in XGBoost. Once auto-tuning is complete, the users are able to get access to a prediction profiler, which allows them to see the relationship between the predictor variables in their dataset, as well as the response that they're modeling. One important aspect of developing an add-in is actually testing it, thoroughly testing it to make sure that it is functioning as expected. What we wanted to do here was, with this auto-tuning functionality enabled, we wanted to test that the prediction profiler was giving the correct numerical output. But this testing… Because we want to make sure that the software is functioning as it should be, so that when users use it, they can have more trust in it.

One of the testing challenges that came up when we were testing this add-in was that there's a large input space to this system. You imagine here in the auto-tuning feature, there's 10 settings. The settings, 8 of them take on this interval-type data where you specify an interval to search for the best hyperparameter settings. On top of that, given a dataset, the prediction profiler can also have numerous predictor variables. The input space for this testing problem is pretty large because there's a lot of different values that users can put into this system. On top of that, a very unique challenge for this testing problem is that the testing time can take quite a while. Depending on what the users specify for their auto-tuning settings, and depending on the dataset they have, auto-tuning could take a long time. It could take 10 minutes or maybe 15 minutes, depending on the settings. If you have any experience with design of experiments, you can kind of think of these auto-tuning settings, all these settings associated with the auto-tuning procedure, as hard-to-change factors or inputs in this testing problem. Whereas the factors and inputs for the prediction profiler are a lot more easy to change. They don't take much time at all because the calculation is very quick.

Lastly, one last challenge that we had was actually determining the oracle. Whenever you're testing a software system or any type of system, you need to know what the expected result or expected output of the system is as a baseline to test what has been implemented in practice. How do we actually… You've seen the system here, we had the total of 27 different factors or inputs, and we classified… The hard-to-change factors were all associated with the auto-tuning procedure, and the easy-to-change factors were all associated with the prediction profiler. Now, one thing I'll say is to actually approach this testing problem, what we did is we decided to perform testing using the dataset that comes with the XGBoost add-in, which is this powder metallurgy dataset. That has 17 predictor variables, so those will be associated with the prediction profiler.

To handle the large input space, what we did was we preselected certain values of each of the inputs to test, and we decided to use covering arrays, which are available in JMP, to help us select which input combinations to actually test. That part can be directly solved using JMP. But the challenge that gave us… The one challenging thing was the long testing time. Directly using covering arrays from JMP, it may be the case that we would… The covering arrays give us a set of test cases to test, but it may be the case that the settings for the auto-tuning procedure are going to change across every single test case. That could take a lot of time for testing, resetting those auto-tuning parameters every single time. What we did is we took a little inspiration from design of experiments, and we decided to create covering arrays that have a split-plot structure. If you know from design of experiments, when you have hard-to-change factors, you often use split-plot designs. For this testing case, we decided to use covering arrays with a split-plot structure to minimize the number of times that we have to reset the auto-tuning settings. Lastly, for the oracle, we decided to use the prediction formula, which can be added as a column in the dataset. We use that prediction formula as a pseudo-oracle so that we can test the output of the prediction profiler against the output of the prediction formula just to make sure that those two values are close together for whatever settings of the prediction profiler we test.

In my previous slide, I mentioned for testing, we wanted to use covering arrays. I'll go a little bit more in depth on what covering arrays are. What we did for our testing problem is we identified combinatorial testing as a suitable framework for approaching our testing problem. Combinatorial testing is this testing method often used to find faults in a system under test by checking if any t-way combinations of system inputs or factors induces a failure for some chosen value of t. This is typically achieved by selecting a suite of test cases that cover all possible t-way combinations of the input settings. Just to visualize what I'm saying here, here in the second image to the right, I have a suite of tests. There's a simple testing problem with three factors, each having two levels. For each test case, what you would do is take the test case, evaluate it under the system under test, get the output, and compare it to your oracle value and see if it's a pass or a fail. Given this data from a test suite and the pass-failure data for each test case, we can investigate which t-way combinations are potentially causing failures in our system so that we can investigate those further and correct them.

This combinatorial testing technique, it often relies heavily on the use of a particular type of design called covering arrays. What covering arrays are, they're a certain type of array, they're available in JMP, and they are an array with N rows and K columns. They satisfy this property that for any selection of t columns… If you specify parameter t, for any selection of t columns, all t-way combinations of those symbols from those columns appear at least once. This is often applied… Covering arrays are often applied in combinatorial testing because they give you a way to cover all t-way combinations of the input settings so that you're really covering the entire test space in a strategic manner. We often want to find covering arrays with a small number of test cases because testing can be a very time-consuming procedure. I have an example of a covering array here, and the bottom image in the center here, this is a strength-two covering array on five factors. The first factor, X1, has three levels, and the remaining just have two levels each. This is a strength-two covering array because for any two columns I pick, all possible two-way combinations appear at least once.

To actually create a covering array in JMP, it's pretty simple. You can just go to DOE, Special Purpose, and go to the Covering Array platform. The user interface looks a lot similar to other DOE platforms, but you can specify the strength t in the Covering Array platform. Once you created a covering array and gather some data on the failure or the pass or fail data for each test case, you can easily analyze it using this Analyze button, and it'll tell you which t-way combinations are likely to be causing a failure in your system. Those are covering arrays and combinatorial testing. These are what are going to be what we're going to use heavily for actually trying to solve our testing problem for the XGBoost add-in. We're going to use these covering arrays to guide us on what we should test. But if you remember, there was a very particular challenge that we faced when testing XGBoost, in particular testing the auto-tuning procedure. If you remember, in the XGBoost testing problem we have, you first do auto-tuning, which can be very time-consuming. Then once you complete the auto-tuning, you get access to the prediction profiler, and you can test different values of the predictor variables and check the predicted response against our pseudo-oracle.

Now, using just traditional covering arrays, it might be the case that we would have to reset the auto-tuning settings numerous times, which could be very time-consuming. What we did is we came up with a way to create covering arrays with split-plot structure so that we could reduce the time spent on testing. We basically wanted to try to minimize the number of times we are changing the auto-tuning procedure settings while preserving the coverage of the test suite. How we do that… How we actually do that is we first identify which system inputs or factors are hard to change and easy to change. Like before, we knew that everything associated with auto-tuning was hard to change, and everything associated with the prediction profiler was easy to change. What we did next is we created a covering array for the hard-to-change factors. That gave us a… We created a strength-two covering array for the hard-to-change factors, which were the auto-tuning settings in the XGBoost testing problem. Then what we did next is we created a covering array for the easy-to-change factors. The factors associated with the prediction profiler, as well as this pseudo factor that we introduced.

The pseudo factor has the same number of levels as the number of runs in the covering array from step two. This pseudo factor will be used for constructing whole plots, which I'll describe what that means in step four here. In step four, once we have created this covering array on the easy-to-change factors and the pseudo factor, what we'll do is we'll create a new data table in JMP and copy over the response and pseudo factor columns from step three. We'll also add columns for all the hard-to-change factors next to the pseudo factor column, and then we'll copy over the easy-to-change factor columns and all their values from step three. What we do next is, depending on the level of the pseudo factor… I did forget to mention in step three, we sort the data table with respect to the pseudo factor to form these nice blocks or whole plots. But going back to step four, what we do is we look at the pseudo factor column, and what we'll do is if we see… Each level of the pseudo factor column is associated with some run in the covering array created in step two. If we see a level one for the pseudo factor in a certain row, we will place the run from the covering array created on the hard-to-change factors in the columns associated with the hard-to-change factors in this larger covering array.

You can see that here, after we create this… After we do this expansion of the pseudo factors, you can see that this array here has a split-plot structure. You can see that the hard-to-change factors, from max depth to number of inner folds, they are held constant across a certain number of runs. That's our whole plots there. Within those whole plots, the easy-to-change factors are allowed to vary freely. How we would use this test suite in practice is we would go to our XGBoost add-in. For the first whole plot, we would put in these settings marked in the big red square here and let the auto-tuning procedure proceed. Then once we're done with auto-tuning, we'd go to the prediction profiler and then test these five test cases, starting from formation method down to the last predictor, P5. This is what we did in JMP 19. But coming in JMP 20, we actually built this procedure into the Covering Array platform in that users can now specify which inputs or factors are hard to change, and we automatically create a covering array that has that split-plot structure while satisfying the user's coverage requirements.

Closing off, going back to the actual testing problem, what we wanted to do is we wanted to show that for testing problems that have these hard-to-change factors, you can really benefit from taking advantage of that structure. You can really take advantage of the presence of hard-to-change factors if you create a test suite that takes that into account. To actually showcase that, what we did for our XGBoost testing problem is we created two covering arrays. One covering array was a traditional covering array that does not take into account the presence of hard-to-change factors. That covering array, so it had 27… There's 27 factors. That covering array was able to… We were able to create a covering array with only 32 test cases using the traditional method. But because that covering array didn't take into account the presence of hard-to-change factors, those hard-to-change factors were reset 32 times. Whereas in our method, we create a covering array with split-plot structure. That covering array had 75 test cases. But because of the fact that we took advantage of the hard-to-change factors, we only have to reset the hard-to-change factors, those auto-tuning settings, only 15 times.

What does this mean? First, using our method, we saw that we actually had higher coverage of the testing space. By construction, both of those covering arrays were strength-two. They were covering all two-way interactions between the input settings. Our covering array with a split-plot structure, it had higher three-way coverage as well as four-way coverage compared to the more traditional covering array that doesn't take into account the hard-to-change factors. A very important advantage we saw was in the testing time. The traditional covering array, the total time it took to use that to go through all the test cases that it created, it took 104 minutes, and that's because we had to reset the auto-tuning settings 32 times. But our covering array with the split-plot structure, we were able to complete testing in just 61 minutes, and that was because we only reset the hard-to-change factors 15 times. In the background, the way we were doing testing, we were doing automated testing. There wasn't much of a… I wasn't manually going through each test case and testing it on my own. I was using JSL scripting to do a lot of the testing procedure. Now, even after adjusting for outliers in the auto-tuning times between these two test suites, the traditional covering array took 77 minutes to complete, while ours took 45 minutes.

We saw both an advantage. By taking into account that there's hard-to-change factors and creating a covering array that had split-plot structure, we were able to save quite a bit of time in testing. On top of that, it also aided in fault localization. What we were doing when we were testing was we would get the value from the prediction profiler and compare that against the value of the prediction formula in the data table. We were comparing them in terms of the number of significant digits where they matched. Of course for the first… They matched up to five significant digits across all the test cases. One interesting thing was that when you compare up to seven significant digits, we saw five failures coming from the test suite associated with the covering array with a split-plot structure. Something interesting there was that those five failures, each one was in a separate whole plot. What that told us was that it looks like there wasn't any failures being induced due to two-way interactions from the hard-to-change factors. That gave us a little bit more confidence in the auto-tuning abilities… In the implementation of the auto-tuning procedure in XGBoost. These other settings, checking the two numbers up to 8, 9, and 10 significant digits, we did see a lot of failures across that case. But I think that's pretty reasonable in that at some point, you're not going to have all of the digits between the pseudo-oracle and oracle matching exactly.

That is my talk here today. In conclusion, what we found was that in some testing problems, certain inputs or factors may be hard to change or reset in every test case. In that case, we proposed a way to construct covering arrays that account for these hard-to-change factors. These covering arrays have a split-plot structure, which is well known in design of experiments. We provided a way to create these covering arrays. We highlighted a process to create these covering arrays manually in previous versions of JMP. We also highlighted that for JMP 20, there's going to be an easier way to create these covering arrays, more user-friendly way to create these covering arrays with split-plot structure. Lastly, what we found was that accounting for hard-to-change factors in the test suite or the test plan can save a lot of time during testing by decreasing the amount of times the hard-to-change factors need to be reset. If you have a testing problem with hard-to-change factors and you have the right structure, you should really take advantage of that when you're creating your test cases in order to save time. Thank you for coming to my talk today.


Start:
Wed, Oct 21, 2026 04:00 PM EDT
End:
Wed, Oct 21, 2026 04:45 PM EDT
Ped 7
Attachments
0 Kudos