Covering arrays are a powerful design tool that may be used to design test cases to efficiently test deterministic systems. For such systems, a particular input will always generate the same output and, as a result, standard statistical designs are usually inefficient. It turns out that failures in these systems are typically precipitated by certain combinations of just a few factors. A covering array ensures that all combinations of a specified number of factors (called the strength) have been covered with as few test cases as possible. In his recent post, Bradley Jones described covering arrays as the most exciting new DOE feature coming in the upcoming release of JMP. The new Covering Array platform is in JMP Pro 12, and it includes these four great features: 1. The “Optimize” button: The platform tries to construct the smallest possible design, and in some cases the designs are optimal. However, when the design created by the platform is not optimal, the platform enables an “Optimize” button that may be used to further reduce the size of the design. You get to specify the number of optimization iterations. You can cancel optimization at any point, and the optimizer will return the best design found. 2. The “Restrict Factor Level Combinations” control: When defining a design, it is sometimes necessary to restrict certain factor level combinations (i.e., disallowed combinations). For example, suppose you were testing a browser-based application, you would want to make sure that your design did not contain incompatible browser, operating system pairs (say, IE10, OSX). The “Restrict Factor Level Combinations” control provides two ways of specifying such restrictions. The idea is to provide a control that is sufficiently flexible that both experienced as well as casual users are accommodated.You may use JSL (JMP Scripting Language) to define restrictions as a Boolean expression. Correctly specifying restrictions as a Boolean expression can sometimes be challenging. To make this easier, you may use a tool similar to a Data Filter to specify restrictions. Note that designs that restrict factor level combinations may also be optimized. 3. The “Analysis” table script: If you create a design data table, a Response column will be added, and an “Analysis” table script will be available. An outcome for any of these runs can either be pass, fail or missing. If at least one of the outcomes is a failure, then the Analysis script may be used to identify potential factor/level combinations that could have precipitated the failure. This identification problem is often referred to as the “failure localization” problem. 4. The “Load Design” menu option: This option allows you to load a covering array design from a JMP data table. Imagine that you have a design, and you are interested in assessing the coverage properties of the design. Or perhaps you would like to see if the number of runs can be reduced while preserving coverage. As long as your design is in a format that can be opened by JMP as a data table, then you can use this option to load it into the platform. You can even load a design with disallowed combinations as long as the disallowed combinations are specified as a Boolean expression in a “Disallowed Combinations” table property. For more on covering arrays, read Brad’s discussion in his post for this series and Ryan Lekivetz’s post, which describes how he cleverly used covering arrays in a home improvement project. Editor's note: This post is part of a series of previews of JMP 12 written by the people who develop the software.
... View more
